Image recognition method and device, electronic equipment and computer readable storage medium

By extracting visual and depth features and building visual depth fusion features, identifying the target object in the image to be identified and its distance from the camera, the problem of inaccurate identification in the prior art is solved, and the accuracy and stability of recognition are improved .

CN118968111APending Publication Date: 2024-11-15SUZHOU QINGZHOU ZHIHANG INTELLIGENT TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310545091.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-15
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

In the prior art, the object detection method has the problem of inaccurate identification when identifying the distance between the target object in the image to be identified and the target camera, especially the accuracy and stability caused by object occlusion, deformation or light changes in the visual image. Challenged.

Method used

By acquiring the visual features and depth features of the image to be identified, and combining the visual depth fusion features, the target object and its distance from the camera are identified. The specific steps include extracting visual features and depth features, building a loss function for model training, acquiring point cloud data for deep feature recognition, and determining the three-dimensional coordinate position of the target object through visual deep fusion features.

Benefits of technology

It realizes more accurately identifying the target object in the image to be identified and its distance from the camera, improves the accuracy and stability of the recognition, and solves the problem of inaccurate recognition.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118968111A_ABST
    Figure CN118968111A_ABST
Patent Text Reader

Abstract

The invention discloses an image recognition method and device, electronic equipment and a computer readable storage medium. The method comprises the following steps: acquiring a to-be-recognized image; visual features of the to-be-recognized image are extracted, a plurality of visual feature maps of a first channel are obtained, depth features of the to-be-recognized image are extracted, a plurality of depth feature maps of a second channel are obtained, and the visual feature maps and the depth feature maps each comprise a predetermined number of pixel points; determining visual depth fusion features according to the plurality of visual feature maps of the first channel and the plurality of depth feature maps of the second channel; according to the visual depth fusion feature, a target object in the to-be-recognized image and a target distance between the target object and a target camera are recognized, and the target camera is a camera shooting the to-be-recognized image. According to the invention, the technical problem of inaccurate recognition when the distance between the target object in the to-be-recognized image and the target camera is recognized in the prior art is solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image recognition, and in particular to an image recognition method, device, electronic device and computer-readable storage medium. Background Art

[0002] The target detection methods in the related art are challenged in terms of accuracy and stability because objects in visual images are often blocked, deformed, or have lighting changes. Therefore, when using the target detection methods in the related art to identify the distance between the target object in the image to be identified and the target camera, there is a technical problem of inaccurate identification.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] Embodiments of the present invention provide an image recognition method, device, electronic device and computer-readable storage medium to at least solve the technical problem of inaccurate recognition when identifying the distance between a target object in an image to be recognized and a target camera in the related art.

[0005] According to one aspect of an embodiment of the present invention, there is provided an image recognition method, characterized in that it includes: acquiring an image to be recognized; extracting visual features of the image to be recognized to obtain several visual feature maps of a first channel, and extracting depth features of the image to be recognized to obtain several depth feature maps of a second channel; determining a visual depth fusion feature based on the several visual feature maps of the first channel and the several depth feature maps of the second channel; and identifying a target object in the image to be recognized and a target distance between the target object and a target camera based on the visual depth fusion feature, wherein the target camera is a camera that takes the image to be recognized.

[0006] Optionally, in the case where the second channel several depth feature maps are obtained by inputting the image to be identified into a deep feature recognition model, the method further includes: training the sample data in the following manner to obtain the deep feature recognition model: obtaining an initial feature recognition model and a loss function for model training, wherein the loss function is used to represent the difference between the sample depth features and the point cloud depth features corresponding to the sample image to be identified; based on the loss function, training the initial feature model with the sample data to obtain the deep feature recognition model, wherein the sample data includes the sample image to be identified, several sample depth feature maps of the second channel, and several point cloud depth feature maps of the second channel.

[0007] Optionally, the method also includes: constructing the loss function in the following manner: obtaining a number of sample depth feature maps of the second channel corresponding to the sample image to be identified, and a number of point cloud depth feature maps of the second channel corresponding to the point cloud data corresponding to the sample image to be identified; constructing the loss function based on the number of sample depth feature maps of the second channel corresponding to the sample image to be identified and the corresponding number of point cloud depth feature maps of the second channel.

[0008] Optionally, before the depth feature recognition model is obtained by training the initial feature model with the sample data based on the loss function, the method further includes: acquiring initial point cloud data collected by a point cloud device; acquiring point cloud data corresponding to the sample image to be identified based on the sample camera intrinsic parameters, the sample camera extrinsic parameters, the point cloud device parameters corresponding to the point cloud device, and the initial point cloud data; converting the point cloud data based on the sample camera intrinsic parameters and the sample camera extrinsic parameters of the sample camera to obtain several point cloud depth feature maps of the second channel corresponding to the point cloud data.

[0009] Optionally, determining the visual depth fusion feature based on the visual feature maps of the first channels and the depth feature maps of the second channels includes: determining the visual feature to be fused based on the visual feature maps of the first channels, and determining the depth feature to be fused based on the depth feature maps of the second channels, wherein the visual feature to be fused is at least a four-dimensional feature, the visual feature to be fused includes the visual dimension features of the first channels, the depth feature to be fused is at least a four-dimensional feature, the depth feature to be fused includes the depth dimension features of the second channels, and the visual feature to be fused has the same dimension as the depth feature to be fused; multiplying the visual feature to be fused with the depth feature to be fused to obtain the visual depth fusion feature, wherein the visual feature to be fused has the same dimension as the visual depth fusion feature, and the visual depth fusion feature includes the visual dimension features of the first channels and the depth dimension features of the second channels.

[0010] Optionally, identifying the target object in the image to be identified and the target distance between the target object and the target camera based on the visual depth fusion feature includes: determining the three-dimensional coordinate positions of object points corresponding to a predetermined number of pixels in the image to be identified in a predetermined three-dimensional space based on the visual depth fusion feature, target camera intrinsic parameters and target camera extrinsic parameters of the target camera; identifying the target object in the image to be identified and the target distance between the target object and the target camera based on the visual depth fusion feature and the three-dimensional coordinate positions of object points corresponding to the predetermined number of pixels in the predetermined three-dimensional space.

[0011] Optionally, determining the three-dimensional coordinate positions of object points corresponding to a predetermined number of pixels in the image to be identified in a predetermined three-dimensional space based on the visual depth fusion feature, the target camera intrinsic parameters and the target camera extrinsic parameters of the target camera includes: determining the two-dimensional coordinate positions of the predetermined number of pixels in the image to be identified and the preset depth values ​​corresponding to the object points corresponding to the predetermined number of pixels based on the visual depth fusion feature; determining the three-dimensional coordinate positions of the object points corresponding to the predetermined number of pixels in the predetermined three-dimensional space based on the two-dimensional coordinate positions of the predetermined number of pixels in the image to be identified, the preset depth values ​​corresponding to the object points corresponding to the predetermined number of pixels, the target camera intrinsic parameters and the target camera extrinsic parameters of the target camera.

[0012] Optionally, the determining the three-dimensional coordinate positions of the object points corresponding to the predetermined number of pixels in a predetermined three-dimensional space according to the two-dimensional coordinate positions of the predetermined number of pixels in the image to be identified, the preset depth values ​​corresponding to the object points corresponding to the predetermined number of pixels, the target camera intrinsic parameters and the target camera extrinsic parameters of the target camera, includes: the three-dimensional coordinate positions include an x-coordinate position, a y-coordinate position, and a z-coordinate position, the two-dimensional coordinate positions include first horizontal coordinates and first vertical coordinates of the predetermined number of pixels in the image to be identified, the target camera intrinsic parameters include second horizontal coordinates and second vertical coordinates of the principal point of the target camera on the image plane, and the target camera extrinsic parameters include the horizontal coordinates of the target camera In the case of the horizontal focal length of the target camera and the vertical focal length of the target camera, determine the first difference between the first horizontal coordinates corresponding to the predetermined number of pixels and the horizontal focal length, and the second difference between the first vertical coordinates corresponding to the predetermined number of pixels and the vertical focal length; determine the first product of the preset depth values ​​corresponding to the object points corresponding to the predetermined number of pixels and the corresponding first difference, and the second product of the preset depth values ​​corresponding to the object points corresponding to the predetermined number of pixels and the corresponding second difference; determine the ratio of the first product to the second horizontal coordinate to be the x-coordinate position, determine the ratio of the second product to the second vertical coordinate to be the y-coordinate position, and determine the predetermined depth value to be the z-coordinate position.

[0013] Optionally, after identifying the target object in the image to be identified and the target distance between the target object and the target camera based on the visual depth fusion feature, the method further includes: when the target distance is less than a predetermined distance, sending a predetermined control instruction to a predetermined terminal, wherein the predetermined control instruction includes at least one of the following: a braking instruction, a deceleration instruction, and a steering instruction.

[0014] According to one aspect of an embodiment of the present invention, an image recognition device is provided, characterized in that it includes: an acquisition module, used to acquire an image to be recognized; an extraction module, used to extract visual features of the image to be recognized, obtain several visual feature maps of the first channel, and extract depth features of the image to be recognized, obtain several depth feature maps of the second channel; a determination module, used to determine visual depth fusion features based on the several visual feature maps of the first channel and the several depth feature maps of the second channel; and an identification module, used to identify a target object in the image to be recognized and a target distance between the target object and a target camera based on the visual depth fusion features, wherein the target camera is a camera that shoots the image to be recognized.

[0015] According to one aspect of an embodiment of the present invention, there is provided an electronic device, comprising: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the following steps: acquiring an image to be identified; extracting visual features of the image to be identified to obtain a number of visual feature maps of a first channel, and extracting depth features of the image to be identified to obtain a number of depth feature maps of a second channel; determining a visual depth fusion feature based on the number of visual feature maps of the first channel and the number of depth feature maps of the second channel; identifying a target object in the image to be identified and a target distance between the target object and a target camera based on the visual depth fusion feature, wherein the target camera is a camera that takes the image to be identified.

[0016] According to one aspect of an embodiment of the present invention, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute any one of the above-mentioned image recognition methods.

[0017] In an embodiment of the present invention, an image to be identified is obtained, and visual features of the image to be identified are extracted to obtain several visual feature maps of the first channel, wherein the visual feature map represents features related to the category of the object in the image to be identified, and whether the object in the image to be identified is the target object can be identified based on this. And the depth features of the image to be identified are extracted to obtain several depth feature maps of the second channel, wherein the depth feature map represents features related to the distance between the object in the image to be identified and the camera, and the distance between the object in the image to be identified and the camera can be identified based on this. Based on several visual feature maps of the first channel and several depth feature maps of the second channel, the visual depth fusion feature is determined. Among them, the visual depth fusion feature includes several visual dimension features of the first channel and several depth dimension features of the second channel, that is, it can represent both the category of the object and the distance between the object and the camera. Then, based on the visual depth fusion feature, the purpose of identifying the target object in the image to be identified and the target distance between the target object and the target camera can be achieved. Since the visual depth fusion feature can represent both the category of the object and the distance between the object and the camera, it is possible to identify the target object in the image to be identified and the target distance between the target object and the target camera through collaborative recognition, thereby achieving the effect of more accurately identifying the target object in the image to be identified and the target distance between the target object and the target camera, thereby solving the technical problem of inaccurate recognition when identifying the distance between the target object in the image to be identified and the target camera in the related technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0019] Figure 1 is a flowchart of an image recognition method according to an embodiment of the present invention;

[0020] Figure 2 is a structural block diagram of an image recognition device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0022] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0023] Example 1

[0024] According to an embodiment of the present invention, an embodiment of an image recognition method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0025] Figure 1 is a flow chart of an image recognition method according to an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:

[0026] Step S102, obtaining an image to be recognized;

[0027] In step S102 provided in the present application, the image to be identified can be an image or a video, or an extracted image frame. The method of obtaining the image to be identified can be set differently according to different applications and scenarios. For example, in the scenario of autonomous driving, the image can be obtained by taking pictures with a camera in the autonomous driving vehicle, and objects that are easy to appear on the road section, such as pedestrians, can be identified.

[0028] Step S104, extracting visual features of the image to be identified to obtain a plurality of visual feature maps of the first channel, and extracting depth features of the image to be identified to obtain a plurality of depth feature maps of the second channel;

[0029] In step S104 provided in the present application, the visual feature map represents features related to the category of the object in the image to be identified, and it can be used to identify whether the object in the image to be identified is the target object. The first channel number of visual feature maps can be represented by (c, H, W), where c represents the number of channels, W represents the width of the image to be identified, and H represents the height of the image to be identified. If the image to be identified is an image of 128*128 pixels, then W is 128, H is also 128, and c can be customized according to the actual application and scene. For example, if it is set to 64, 64 represents 64 channels. In each layer of channels, different visual features can be represented, such as color, etc. This reflects the characteristics of the target object.

[0030] The depth feature map represents the features related to the distance between the object in the image to be identified and the camera, and can be used to identify the distance between the object in the image to be identified and the camera. The second channel depth feature map can be represented by (c, H, W), where c represents the number of channels, W represents the width of the image to be identified, and H represents the height of the image to be identified. If the image to be identified is an image of 128*128 pixels, then W is 128 and H is also 128. c can be customized according to the actual application and scenario. For example, if it is set to 57, 57 represents 57 channels. 57 can mean that there are 57 depth values ​​at a depth interval of (4, 60) meters. The value in each layer of the channel can be expressed as the probability that the object point corresponding to the pixel point may be at each depth value. This reflects the characteristics of the distance.

[0031] Optionally, in the process of extracting the visual features of the image to be identified and obtaining several visual feature maps of the first channel, the extraction can be performed in a variety of ways, for example, the extraction can be performed by a neural network model, such as inputting the image to be identified into a visual feature recognition model to obtain several visual feature maps of the first channel, wherein the visual feature recognition model is obtained through training with sample data, and the sample data includes a sample image to be identified, a sample object in the sample image to be identified, and a sample distance between the sample object and a sample camera;

[0032] Also optionally, in the process of extracting the depth features of the image to be identified and obtaining several depth feature maps of the second channel, the extraction can be performed in a variety of ways. For example, the extraction can also be performed by a neural network model. For example, the image to be identified can be input into a deep feature recognition model to obtain several depth feature maps of the second channel.

[0033] It should be noted that in the optional method provided in the present application, an object recognition neural network model is used. Since the object recognition neural network model is obtained by training the initial model with sample data, the use of the trained model can better recognize the image to be recognized, and can make the recognition result more accurate and faster.

[0034] It should also be noted that in the optional method provided in the present application, the visual feature recognition model can be a model composed of a network formed by combining a residual network and a feature pyramid network. By combining these two networks to identify features related to the category of the object in the image to be identified, the technical effect of more accurately and quickly determining several visual feature maps of the first channel can be achieved.

[0035] Step S106, determining a visual depth fusion feature according to a plurality of visual feature maps of the first channel and a plurality of depth feature maps of the second channel;

[0036] In step S106 provided in the present application, the visual features and the depth features are fused to obtain higher-dimensional visual depth fusion features. It should be noted that when identifying the image to be identified, if several feature maps of predetermined channels related to other aspects are also to be identified, at this time, feature fusion is performed, and five-dimensional or even higher-dimensional visual depth fusion features may be obtained. Through higher-dimensional visual depth fusion features, the features of the two can be fused with each other, and the subsequent target object and the distance between the target object and the target camera can be jointly judged, which increases the recognition accuracy of this method to a certain extent.

[0037] Step S108 , identifying the target object in the image to be identified and the target distance between the target object and the target camera based on the visual depth fusion feature, wherein the target camera is the camera that shoots the image to be identified.

[0038] In step S108 provided in the present application, the target object in the image to be identified and the target distance between the target object and the target camera are identified through visual depth fusion features, thereby achieving the purpose of identifying the target object in the image to be identified and the target distance between the target object and the target camera.

[0039] Through the above steps, the image to be identified is obtained, the visual features of the image to be identified are extracted, and several visual feature maps of the first channel are obtained, wherein the visual feature map represents the features related to the category of the object in the image to be identified, and it can be used to identify whether the object in the image to be identified is the target object. And the depth features of the image to be identified are extracted to obtain several depth feature maps of the second channel, wherein the depth feature map represents the features related to the distance between the object in the image to be identified and the camera, and it can be used to identify the distance between the object in the image to be identified and the camera. According to the several visual feature maps of the first channel and the several depth feature maps of the second channel, the visual depth fusion features are determined. Among them, the visual depth fusion features include several visual dimension features of the first channel and several depth dimension features of the second channel, that is, they can represent both the category of the object and the distance between the object and the camera. Then, according to the visual depth fusion features, the purpose of identifying the target object in the image to be identified and the target distance between the target object and the target camera can be achieved. Since the visual depth fusion feature can represent both the category of the object and the distance between the object and the camera, it is possible to identify the target object in the image to be identified and the target distance between the target object and the target camera through collaborative recognition, thereby achieving the effect of more accurately identifying the target object in the image to be identified and the target distance between the target object and the target camera, thereby solving the technical problem of inaccurate recognition when identifying the distance between the target object in the image to be identified and the target camera in the related technology.

[0040] As an optional embodiment, when several depth feature maps of the second channel are obtained by inputting the image to be identified into a deep feature recognition model, the method also includes: training the sample data in the following manner to obtain a deep feature recognition model: obtaining an initial feature recognition model and a loss function for model training, wherein the loss function is used to represent the difference between the sample depth features and the point cloud depth features corresponding to the sample image to be identified; based on the loss function, training the initial feature model with the sample data to obtain a deep feature recognition model, wherein the sample data includes the sample image to be identified, several sample depth feature maps of the second channel, and several point cloud depth feature maps of the second channel.

[0041] In this embodiment, the case where the image to be identified is input into the deep feature recognition model to obtain several deep feature maps of the second channel is described. In this case, an initial feature recognition model and a loss function for model training are obtained. The loss function is used to represent the difference between the sample deep features corresponding to the sample image to be identified and the point cloud deep features. Based on the loss function, the initial feature model is trained with sample data to obtain a deep feature recognition model.

[0042] It should be noted that during the training process, a loss function is used to supervise the depth feature maps of several samples in the second channel of the depth prediction output. Since the loss function is used to characterize the difference between the sample depth features and the point cloud depth features corresponding to the sample image to be identified. The point cloud depth features are more capable of better reflecting the depth information. Therefore, by characterizing the difference between the two through the loss function, the initial feature recognition model is trained to obtain the depth feature recognition model, which can make the depth feature map identified by the obtained depth feature recognition model more accurate. That is, through the supervision of point cloud data, the accuracy of the obtained second channel several depth feature maps is improved. In this way, not only can the accuracy of the second channel several depth feature maps be greatly improved. Moreover, the supervision is only in the training process. During the use of the model, there is no supervision, which does not affect the application reasoning speed of the model. In actual use, there is no need to deploy equipment related to point cloud data, and there is no need to use additional sensors or multiple cameras for data collection. An RGB image can achieve accurate recognition, which speeds up the recognition speed of the target object and saves computing resources and deployment costs.

[0043] As an optional embodiment, a loss function is constructed in the following manner: obtaining a number of sample depth feature maps of the second channel corresponding to the sample image to be identified, and a number of point cloud depth feature maps of the second channel corresponding to the point cloud data corresponding to the sample image to be identified; based on the number of sample depth feature maps of the second channel corresponding to the sample image to be identified and the corresponding number of point cloud depth feature maps of the second channel, a loss function is constructed.

[0044] In this embodiment, it is explained that the loss function is used to characterize the depth feature map of several samples in the second channel corresponding to the sample image to be identified, and the depth feature map of several point clouds in the second channel corresponding to the point cloud data corresponding to the sample image to be identified is constructed, which reflects the difference between the depth feature map of several samples in the second channel corresponding to the sample image to be identified and the depth feature map of several point clouds in the second channel corresponding to the point cloud data corresponding to the sample image to be identified. Since the point cloud data is more capable of better reflecting the depth information, the difference between the two is characterized by the loss function, so that the initial feature recognition model is trained, and the trained depth feature recognition model obtained can make the obtained depth feature map more accurate. That is, through the supervision of the point cloud data, the accuracy of the obtained second channel several depth feature maps is improved. In this way, the accuracy of the second channel several depth feature maps can be greatly improved.

[0045] As an optional embodiment, based on the loss function, the initial feature model is trained with sample data to obtain a deep feature recognition model, and it also includes: obtaining initial point cloud data collected by a point cloud device; obtaining point cloud data corresponding to the sample image to be identified based on the sample camera internal parameters, the sample camera external parameters, the point cloud device parameters corresponding to the point cloud device, and the initial point cloud data; converting the point cloud data based on the sample camera internal parameters and the sample camera external parameters of the sample camera to obtain several point cloud depth feature maps of the second channel corresponding to the point cloud data.

[0046] In this embodiment, in the training step before the initial feature model is trained with sample data based on the loss function to obtain the deep feature recognition model, it is necessary to obtain several point cloud depth feature maps of the second channel. In the process of obtaining several point cloud depth feature maps of the second channel, it is first necessary to obtain the initial point cloud data collected by the point cloud device, wherein the point cloud device can be a device that can obtain point cloud data, such as a depth camera. Since the initial point cloud data collected by the point cloud device may not correspond to the image to be recognized, for example, the point cloud data and the image to be recognized may have an angle deviation due to different camera setting positions. Therefore, it is necessary to obtain the internal and external parameters of the sample camera, as well as the point cloud parameters corresponding to the point cloud device, to correct the initial point cloud data, and obtain point cloud data that can correspond to the sample image to be recognized. This makes the point cloud data more accurate and can better reflect the depth features.

[0047] After obtaining the point cloud data corresponding to the sample image to be identified, the point cloud data can be converted into a plurality of point cloud depth feature maps of the second channel according to the sample camera intrinsic parameters and sample camera extrinsic parameters of the sample camera. That is, the point cloud data generates a point cloud depth feature map and uses it for supervision.

[0048] Specifically, the point cloud data can be first converted into the depth data of the image through the sample camera internal parameters and sample camera external parameters of the sample camera, and then converted into a depth distribution, where the probability value at the depth value position is 1 and the other positions are 0, and finally the second channel number of point cloud depth feature maps are generated. When the second channel number is 57, the second channel number of sample depth feature maps (57, H, W) can be generated. Through the supervision of the point cloud data, the accuracy of the obtained second channel number of depth feature maps can be greatly improved.

[0049] As an optional embodiment, the visual depth fusion feature is determined based on several visual feature maps of the first channel and several depth feature maps of the second channel, including: determining the visual feature to be fused based on several visual feature maps of the first channel, and determining the depth feature to be fused based on several depth feature maps of the second channel, wherein the visual feature to be fused is at least a four-dimensional feature, the visual feature to be fused includes several visual dimension features of the first channel, the depth feature to be fused is at least a four-dimensional feature, the depth feature to be fused includes several depth dimension features of the second channel, and the dimension of the visual feature to be fused is the same as that of the depth feature to be fused; multiplying the visual feature to be fused with the depth feature to be fused to obtain the visual depth fusion feature, wherein the visual feature to be fused has the same dimension as the visual depth fusion feature, and the visual depth fusion feature includes several visual dimension features of the first channel and several depth dimension features of the second channel.

[0050] In this embodiment, it is explained how to obtain the visual depth fusion feature. First, based on the first channel several visual feature maps, the visual features to be fused with the same dimension as the visual depth fusion feature are determined, and based on the second channel several depth feature maps, the depth features to be fused with the same dimension as the visual depth fusion feature are determined. In this way, the visual features to be fused and the depth features to be fused can be operated in the same dimension to obtain the visual depth fusion feature.

[0051] For example, assuming that the visual feature maps of the first channel are represented as (64, H, W), and the depth feature maps of the second channel are represented as (57, H, W), the visual features to be fused (64, 1, H, W) are determined based on the visual feature maps of the first channel, and the depth features to be fused (1, 57, H, W) are determined based on the depth feature maps of the second channel. The visual features to be fused (64, 1, H, W) are multiplied with the depth features to be fused (1, 57, H, W) to obtain the visual depth fusion features (64, 57, H, W), thereby achieving the purpose of obtaining the visual depth fusion features.

[0052] It should be noted that the visual features and the depth features are fused to obtain higher-dimensional visual depth fusion features, that is, visual depth fusion features of at least four-dimensional features. When identifying the image to be identified, if several feature maps of predetermined channels related to other aspects are also to be identified, at this time, feature fusion is performed, and five-dimensional or even higher-dimensional visual depth fusion features may be obtained. Through higher-dimensional visual depth fusion features, multiple features can be fused with each other to jointly judge the subsequent target object and the distance between the target object and the target camera, which increases the recognition accuracy of this method to a certain extent.

[0053] As an optional embodiment, based on the visual depth fusion feature, the target object in the image to be identified and the target distance between the target object and the target camera are identified, including: determining the three-dimensional coordinate positions of object points corresponding to a predetermined number of pixels in the image to be identified in a predetermined three-dimensional space based on the visual depth fusion feature, the target camera intrinsic parameters and the target camera extrinsic parameters of the target camera; based on the visual depth fusion feature and the three-dimensional coordinate positions of object points corresponding to a predetermined number of pixels in the predetermined three-dimensional space, the target object in the image to be identified and the target distance between the target object and the target camera.

[0054] In this embodiment, the steps of identifying the target object in the image to be identified and the target distance between the target object and the camera are described. In this step: through the visual depth fusion feature, the target camera internal parameters and the target camera external parameters of the target camera, the three-dimensional coordinate positions of the object points corresponding to the predetermined number of pixels in the image to be identified in the predetermined three-dimensional space are determined. If the pixels of the image to be identified are 128*128 pixels, the predetermined number of pixels are 128*128 pixels, and the object points corresponding to these 128*128 pixels, that is, the corresponding objects in the predetermined three-dimensional space, are determined. Among them, the setting of the coordinate system of the three-dimensional coordinate position can be customized according to the actual application and scenario. For example, in the autonomous driving scenario, the camera in the autonomous driving vehicle can be used as the origin to establish a three-dimensional space rectangular coordinate system of the xyz axis, and so on.

[0055] Then, based on the visual depth fusion feature and the three-dimensional coordinate positions of the object points corresponding to the predetermined number of pixels in the predetermined three-dimensional space, the target object in the image to be identified and the target distance between the target object and the camera are identified. Since the three-dimensional coordinate positions of the object points corresponding to the predetermined number of pixels in the predetermined three-dimensional space are taken into consideration, the distance is determined more accurately, thereby achieving the purpose of more accurately identifying the target object in the image to be identified and the target distance between the target object and the camera.

[0056] As an optional embodiment, the three-dimensional coordinate positions of object points corresponding to a predetermined number of pixels in the image to be identified in a predetermined three-dimensional space are determined based on visual depth fusion features, target camera intrinsic parameters and target camera extrinsic parameters of the target camera, including: determining the two-dimensional coordinate positions of a predetermined number of pixels in the image to be identified and the preset depth values ​​corresponding to the object points corresponding to the predetermined number of pixels based on visual depth fusion features; determining the three-dimensional coordinate positions of object points corresponding to a predetermined number of pixels in the predetermined three-dimensional space based on the two-dimensional coordinate positions of the predetermined number of pixels in the image to be identified, the preset depth values ​​corresponding to the object points corresponding to the predetermined number of pixels, the target camera intrinsic parameters and the target camera extrinsic parameters of the target camera.

[0057] In this embodiment, the steps of determining the three-dimensional coordinate positions of the object points corresponding to a predetermined number of pixels in the image to be identified in a predetermined three-dimensional space are described. In this step, first, the two-dimensional coordinate positions of the predetermined number of pixels in the image to be identified, that is, the positions of the pixels in the image to be identified, and the preset depth values ​​corresponding to the object points corresponding to the predetermined number of pixels, that is, the preset depth values ​​corresponding to the objects corresponding to the pixels in the predetermined three-dimensional space are determined based on the visual depth fusion feature. Based on the two-dimensional coordinate positions of the predetermined number of pixels in the image to be identified, the preset depth values ​​corresponding to the object points corresponding to the predetermined number of pixels, the target camera internal parameters and the target camera external parameters, the three-dimensional coordinate positions of the object points corresponding to the predetermined number of pixels in the predetermined three-dimensional space are determined. In this way, the determination of the three-dimensional coordinate position can be made more accurate.

[0058] As an optional embodiment, according to the two-dimensional coordinate positions of a predetermined number of pixel points in the image to be identified, the preset depth values ​​corresponding to the object points corresponding to the predetermined number of pixel points, the target camera intrinsic parameters and the target camera extrinsic parameters of the target camera, the three-dimensional coordinate positions of the object points corresponding to the predetermined number of pixel points in the predetermined three-dimensional space are determined, including: the three-dimensional coordinate positions include an x-coordinate position, a y-coordinate position, and a z-coordinate position, the two-dimensional coordinate positions include first horizontal coordinates and first vertical coordinates of the predetermined number of pixel points in the image to be identified, the target camera intrinsic parameters include second horizontal coordinates and second vertical coordinates of the principal point of the target camera on the image plane, and the target camera extrinsic parameters include the target camera In the case of the transverse focal length in the horizontal direction and the longitudinal focal length of the target camera in the vertical direction, determine the first difference between the first horizontal coordinate corresponding to the predetermined number of pixels and the transverse focal length, and the second difference between the first vertical coordinate corresponding to the predetermined number of pixels and the longitudinal focal length; determine the first product of the preset depth values ​​corresponding to the object points corresponding to the predetermined number of pixels and the corresponding first difference, and the second product of the preset depth values ​​corresponding to the object points corresponding to the predetermined number of pixels and the corresponding second difference; determine the ratio of the first product to the second horizontal coordinate as the x-coordinate position, determine the ratio of the second product to the second vertical coordinate as the y-coordinate position, and determine the predetermined depth value as the z-coordinate position.

[0059] In this embodiment, a specific calculation method is illustrated, which can be determined by the calculation shown in the following formula:

[0060]

[0061] The three-dimensional coordinate position of the object point corresponding to each pixel point can be calculated by the above formula. Among them, u, v represent the two-dimensional coordinate positions of the predetermined number of pixel points in the image to be identified, that is, the first horizontal coordinate and the first vertical coordinate of the predetermined number of pixel points in the image to be identified, fx, fy, Cx, Cy, represent the target camera intrinsic parameters and target camera extrinsic parameters of the target camera, that is, the second horizontal coordinate and the second vertical coordinate of the principal point of the target camera on the image plane included in the above target camera intrinsic parameters, the target camera extrinsic parameters include the horizontal focal length of the target camera in the horizontal direction, and the longitudinal focal length of the target camera in the vertical direction, and d represents the preset depth values ​​corresponding to the object points corresponding to the predetermined number of pixel points. Since the two-dimensional coordinate positions of the pixel points in the image to be identified, the preset depth values ​​corresponding to the object points corresponding to the pixel points, and the parameters of the camera are considered, comprehensive considerations are made in all aspects and directions, so that the three-dimensional coordinate positions corresponding to the object points corresponding to the predetermined number of pixel points in the predetermined three-dimensional space can be determined to be more accurate.

[0062] In the related art, when identifying the distance between the target object in the image to be identified and the target camera, there is inaccurate recognition. As an optional embodiment, the visual depth fusion feature is input into the object recognition module in the object recognition neural network, and the target object in the image to be identified and the target distance between the target object and the camera are identified, it also includes: when the target distance is less than a predetermined distance, sending a predetermined control instruction to a predetermined terminal, wherein the predetermined control instruction includes at least one of the following: a braking instruction, a deceleration instruction, and a steering instruction.

[0063] In this embodiment, the image recognition method is applied to the field of autonomous driving, and the determination of the target object in the image to be identified and the target distance between the target object and the target camera is realized in the autonomous driving scene. When the target distance is less than the predetermined distance, that is, when the target object is very close to the target camera in the autonomous driving vehicle, the autonomous driving vehicle must be manipulated to avoid the target object. Avoidance can be performed by sending a predetermined control instruction to a predetermined terminal, such as sending a braking instruction to a predetermined terminal to control the autonomous driving vehicle to brake; sending a deceleration instruction to a predetermined terminal to control the autonomous driving vehicle to decelerate; and sending a steering instruction to a predetermined terminal to control the autonomous driving vehicle to turn. Thereby achieving the purpose of avoiding the target object, avoiding the occurrence of traffic accidents, and ensuring driving safety.

[0064] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0065] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods of various embodiments of the present invention.

[0066] Example 2

[0067] According to an embodiment of the present invention, a device for implementing the above-mentioned image recognition method is also provided. Figure 2 is a structural block diagram of an image recognition device according to an embodiment of the present invention. Figure 2 As shown, the device includes: an acquisition module 202, an extraction module 204, a determination module 206 and an identification module 208. The device is described in detail below.

[0068] The acquisition module 202 is used to acquire the image to be identified; the extraction module 204 is used to extract the visual features of the image to be identified to obtain several visual feature maps of the first channel, and extract the depth features of the image to be identified to obtain several depth feature maps of the second channel; the determination module 206 is used to determine the visual depth fusion features based on the several visual feature maps of the first channel and the several depth feature maps of the second channel; the identification module 208 is used to identify the target object in the image to be identified and the target distance between the target object and the target camera based on the visual depth fusion features, wherein the target camera is the camera that shoots the image to be identified.

[0069] It should be noted here that the above-mentioned acquisition module 202, extraction module 204, determination module 206 and recognition module 208 correspond to steps S102 to S108 in implementing the image recognition method, and the instances and application scenarios implemented by the multiple modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned embodiment 1.

[0070] Example 3

[0071] According to another aspect of an embodiment of the present invention, an electronic device is also provided, comprising: a processor; a memory for storing processor executable instructions, wherein the processor is configured to execute instructions to implement the following steps: acquiring an image to be identified; extracting visual features of the image to be identified to obtain several visual feature maps of the first channel, and extracting depth features of the image to be identified to obtain several depth feature maps of the second channel; determining visual depth fusion features based on the several visual feature maps of the first channel and the several depth feature maps of the second channel; identifying a target object in the image to be identified and a target distance between the target object and a target camera based on the visual depth fusion features, wherein the target camera is a camera that takes the image to be identified.

[0072] Example 4

[0073] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device can execute any of the above-mentioned image recognition methods.

[0074] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0075] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0076] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0077] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0078] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0079] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0080] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. An image recognition method, characterized in that: include: Obtain an image to be recognized; Extracting visual features of the image to be identified to obtain a plurality of visual feature maps of the first channel, and extracting depth features of the image to be identified to obtain a plurality of depth feature maps of the second channel; Determining a visual depth fusion feature according to the first channel plurality of visual feature maps and the second channel plurality of depth feature maps; Based on the visual depth fusion feature, a target object in the image to be identified and a target distance between the target object and a target camera are identified, wherein the target camera is a camera that shoots the image to be identified.

2. The method according to claim 1, characterized in that In the case where the second channel plurality of depth feature maps are obtained by inputting the image to be identified into a depth feature recognition model, the method further comprises: training the sample data in the following manner to obtain the depth feature recognition model: Obtaining an initial feature recognition model and a loss function for model training, wherein the loss function is used to represent the difference between the sample depth feature and the point cloud depth feature corresponding to the sample image to be recognized; Based on the loss function, the initial feature model is trained using the sample data to obtain the deep feature recognition model, wherein the sample data includes the sample image to be identified, several sample depth feature maps of the second channel, and several point cloud depth feature maps of the second channel.

3. The method according to claim 2, characterized in that The method further comprises: constructing the loss function by: Acquire a second channel number of sample depth feature maps corresponding to the sample image to be identified, and a second channel number of point cloud depth feature maps corresponding to the point cloud data corresponding to the sample image to be identified; The loss function is constructed based on the second channel number of sample depth feature maps corresponding to the sample image to be identified and the corresponding second channel number of point cloud depth feature maps.

4. The method according to claim 2, characterized in that: Before the initial feature model is trained based on the loss function and the sample data is used to obtain the deep feature recognition model, the method further includes: Acquire initial point cloud data collected by a point cloud device; Acquire point cloud data corresponding to the sample image to be identified according to the sample camera internal parameters, the sample camera external parameters, point cloud device parameters corresponding to the point cloud device, and the initial point cloud data; The point cloud data is converted according to sample camera intrinsic parameters and sample camera extrinsic parameters of the sample camera to obtain a second channel of several point cloud depth feature maps corresponding to the point cloud data.

5. The method according to claim 1, characterized in that Determining the visual depth fusion feature according to the first channel plurality of visual feature maps and the second channel plurality of depth feature maps includes: Determine the visual features to be fused according to the visual feature maps of the first channels, and determine the depth features to be fused according to the depth feature maps of the second channels, wherein the visual features to be fused are at least four-dimensional features, the visual features to be fused include the visual dimension features of the first channels, the depth features to be fused are at least four-dimensional features, the depth features to be fused include the depth dimension features of the second channels, and the visual features to be fused have the same dimension as the depth features to be fused; The visual feature to be fused is multiplied by the depth feature to be fused to obtain the visual depth fusion feature, wherein the visual feature to be fused and the visual depth fusion feature have the same dimension.

6. The method according to claim 1, characterized in that The step of identifying the target object in the image to be identified and the target distance between the target object and the target camera based on the visual depth fusion feature includes: Determine the three-dimensional coordinate positions of object points corresponding to a predetermined number of pixel points in the image to be identified in a predetermined three-dimensional space according to the visual depth fusion feature, the target camera intrinsic parameters and the target camera extrinsic parameters of the target camera; According to the visual depth fusion feature and the three-dimensional coordinate positions of the object points corresponding to the predetermined number of pixels in a predetermined three-dimensional space, the target object in the image to be identified and the target distance between the target object and the target camera are identified.

7. The method according to claim 6, characterized in that The determining, based on the visual depth fusion feature, the target camera intrinsic parameters and the target camera extrinsic parameters of the target camera, the three-dimensional coordinate positions of the object points corresponding to a predetermined number of pixel points in the image to be identified in a predetermined three-dimensional space respectively comprises: Determine, based on the visual depth fusion feature, the two-dimensional coordinate positions of the predetermined number of pixel points in the image to be identified, and the preset depth values ​​corresponding to the object points corresponding to the predetermined number of pixel points; According to the two-dimensional coordinate positions of the predetermined number of pixel points in the image to be identified, the preset depth values ​​corresponding to the object points corresponding to the predetermined number of pixel points, the target camera intrinsic parameters and the target camera extrinsic parameters of the target camera, the three-dimensional coordinate positions of the object points corresponding to the predetermined number of pixel points in the predetermined three-dimensional space are determined.

8. The method according to claim 7, characterized in that The method of determining the three-dimensional coordinate positions of the object points corresponding to the predetermined number of pixels in the image to be identified, the preset depth values ​​corresponding to the object points corresponding to the predetermined number of pixels, the target camera intrinsic parameters and the target camera extrinsic parameters of the target camera, in a predetermined three-dimensional space, comprises: In the case where the three-dimensional coordinate position includes an x-coordinate position, a y-coordinate position, and a z-coordinate position, the two-dimensional coordinate position includes first horizontal coordinates and first vertical coordinates of the predetermined number of pixel points in the image to be identified, the target camera intrinsic parameters include second horizontal coordinates and second vertical coordinates of the principal point of the target camera on the image plane, and the target camera extrinsic parameters include a horizontal focal length of the target camera in the horizontal direction and a vertical focal length of the target camera, determine a first difference between the first horizontal coordinates corresponding to the predetermined number of pixel points and the horizontal focal length, and a second difference between the first vertical coordinates corresponding to the predetermined number of pixel points and the vertical focal length; Determine a first product of the preset depth values ​​corresponding to the object points corresponding to the predetermined number of pixels and the corresponding first difference value, and a second product of the preset depth values ​​corresponding to the object points corresponding to the predetermined number of pixels and the corresponding second difference value; The ratio of the first product to the second abscissa is determined as the x-coordinate position, the ratio of the second product to the second ordinate is determined as the y-coordinate position, and the predetermined depth value is determined as the z-coordinate position.

9. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the following steps: Obtain an image to be recognized; Extracting visual features of the image to be identified to obtain a plurality of visual feature maps of the first channel, and extracting depth features of the image to be identified to obtain a plurality of depth feature maps of the second channel; Determining a visual depth fusion feature according to the first channel plurality of visual feature maps and the second channel plurality of depth feature maps; Based on the visual depth fusion feature, a target object in the image to be identified and a target distance between the target object and a target camera are identified, wherein the target camera is a camera that shoots the image to be identified.

10. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the image recognition method as claimed in any one of claims 1 to 8.