Image recognition processing method and device, equipment and medium

Abstract

The embodiment of the invention relates to an image recognition processing method and device, equipment and a medium, and the method comprises the steps: obtaining a target image collected by a vehicle camera, and obtaining the camera internal reference of the vehicle camera; the target image and the camera internal reference are input into a pre-trained recognition model, and the recognition model comprises a first detection branch head, a second detection branch head and a third detection branch head; and obtaining two-dimensional positioning associated information of the first entity object output by the second detection branch head, three-dimensional positioning associated information of the second entity object output by the third detection branch head, and entity object type information of the first entity object and the second entity object output by the first detection branch head. According to the technical scheme, the method can flexibly adapt to images from different camera parameters to recognize the entity object contained in the driving environment, and the entity object is recognized in combination with two dimensions and three dimensions, so that the robustness and accuracy of entity object recognition are improved.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an image recognition processing method, device, equipment and medium. Background Art

[0002] In the field of autonomous driving, visual perception is one of the key technologies to ensure that the vehicle can correctly understand the driving environment, which involves the identification and positioning of physical objects such as obstacles, pedestrians, and vehicles around the vehicle.

[0003] In the related art, in order to identify physical objects around a vehicle, an image of the vehicle's surroundings is captured by a vehicle-mounted camera, and related physical objects contained in the image are identified based on a pre-trained recognition model.

[0004] However, the above pre-trained recognition model is trained under the assumption that all images follow the same camera intrinsic parameters, which means that when the on-board camera changes, the recognition model may not be able to adapt correctly, resulting in a decrease in the accuracy of identifying physical objects. Summary of the invention

[0005] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides an image recognition processing method, device, equipment and medium, which can flexibly adapt to images from different camera parameters to identify physical objects contained in the driving environment, and combine two-dimensional and three-dimensional to jointly identify physical objects, thereby improving the robustness and accuracy of physical object recognition.

[0006] An embodiment of the present disclosure provides an image recognition processing method, the method comprising: obtaining a target image captured by a vehicle camera, and obtaining camera intrinsic parameters of the vehicle camera; inputting the target image and the camera intrinsic parameters into a pre-trained recognition model, wherein the recognition model comprises a first detection branch head, a second detection branch head, and a third detection branch head; obtaining two-dimensional positioning association information of a first entity object output by the second detection branch head, three-dimensional positioning association information of a second entity object output by the third detection branch head, and entity object type information of the first entity object and the second entity object output by the first detection branch head.

[0007] The disclosed embodiment also provides an image recognition processing device, which includes: a first acquisition module, used to acquire a target image captured by a vehicle camera and acquire the camera intrinsic parameters of the vehicle camera; an input module, used to input the target image and the camera intrinsic parameters into a pre-trained recognition model, wherein the recognition model includes a first detection branch head, a second detection branch head and a third detection branch head; a second acquisition module, used to acquire two-dimensional positioning association information of the first entity object output by the second detection branch head, three-dimensional positioning association information of the second entity object output by the third detection branch head, and entity object type information of the first entity object and the second entity object output by the first detection branch head.

[0008] An embodiment of the present disclosure also provides an electronic device, which includes: a processor; a memory for storing executable instructions of the processor; the processor is used to read the executable instructions from the memory and execute the instructions to implement the image recognition processing method provided by the embodiment of the present disclosure.

[0009] The embodiment of the present disclosure further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is used to execute the image recognition processing method provided by the embodiment of the present disclosure.

[0010] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has the following advantages:

[0011] The image recognition processing scheme provided by the embodiment of the present disclosure obtains the target image captured by the vehicle camera and obtains the camera internal parameters of the vehicle camera, and then inputs the target image and the camera internal parameters into a pre-trained recognition model, wherein the recognition model includes a first detection branch head, a second detection branch head and a third detection branch head, and obtains the two-dimensional positioning association information of the first entity object output by the second detection branch head, the three-dimensional positioning association information of the second entity object output by the third detection branch head, and the entity object type information of the first entity object and the second entity object output by the first detection branch head. In this technical scheme, it is possible to flexibly adapt to images from different camera parameters to recognize entity objects contained in the driving environment, and to combine two-dimensional and three-dimensional to jointly recognize entity objects, thereby improving the robustness and accuracy of entity object recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0013] Figure 1A flowchart of an image recognition processing method provided by an embodiment of the present disclosure;

[0014] Figure 2 A structural diagram of a recognition model provided by an embodiment of the present disclosure;

[0015] Figure 3 A schematic diagram of the structure of another recognition model provided by an embodiment of the present disclosure;

[0016] Figure 4 A schematic diagram of the structure of another recognition model provided by an embodiment of the present disclosure;

[0017] Figure 5 A schematic diagram of the structure of an image recognition processing device provided by an embodiment of the present disclosure;

[0018] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0019] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0020] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0021] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0022] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0023] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0024] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0025] In order to solve the above problems, the embodiment of the present disclosure provides an image recognition processing method, in which, on the one hand, the camera intrinsic parameters and the target image captured by the vehicle-mounted camera are used as inputs, and the changes of the camera intrinsic parameters are considered for visual perception, so as to ensure the accuracy and robustness of recognition. Therefore, while detecting the captured image, the camera intrinsic parameters are also used as part of the model learning, which greatly improves the situation of missed detection of far-field targets and wrong size detection due to the use of the same or fixed intrinsic parameters in the model; on the other hand, considering that in the actual visual perception process, in addition to near-field perception, far-field perception is also included, wherein near-field perception refers to the perception of the area near the vehicle, and far-field perception refers to the perception of the area farther away from the vehicle. The entity objects perceived in the near-field include pedestrians, surrounding vehicles, etc., and the entity objects perceived in the far-field include signal lights, vehicles in front, etc. Near-field perception can have clearer details in the captured image, while far-field perception has insufficient details in the captured image, and the size is also small, and the perception is more difficult. Therefore, in order to fully perceive the entity objects contained in the near field and the far field, the 2D detection head and the 3D detection head can also be combined to identify the entity objects in the captured image. Among them, the 2D detection head can better identify physical objects in the far field, and the 3D detection head can more accurately perceive physical objects in the near field.

[0026] The method is introduced below in conjunction with specific embodiments.

[0027] Figure 1 A flowchart of an image recognition processing method provided in an embodiment of the present disclosure is provided. The method can be executed by an image recognition processing device, wherein the device can be implemented using software and / or hardware and can generally be integrated in an electronic device.

[0028] like Figure 1 As shown, the method includes:

[0029] Step 101, obtaining a target image captured by a vehicle camera, and obtaining the camera intrinsic parameters of the vehicle camera.

[0030] Among them, camera intrinsic parameters can include key parameters such as focal length and principal point position, which are crucial for understanding the actual size and distance of physical objects in the target image.

[0031] In some possible embodiments, the model of the vehicle-mounted camera may be queried, and the camera internal parameters may be obtained by querying a preset database based on the model.

[0032] Step 102: input the target image and the camera internal parameters into a pre-trained recognition model, wherein the recognition model includes a first detection branch head, a second detection branch head, and a third detection branch head.

[0033] In one embodiment of the present disclosure, sample images may be collected in advance, wherein the sample images are not only collected using different vehicle-mounted cameras, but also collected using vehicle-mounted cameras with at least two camera intrinsic parameters, long focal length and short focal length, for the same driving scene. A recognition model is trained based on the sampled sample images, wherein during the training process, the first sample entity object and the second sample entity object, the two-dimensional positioning association information of the first sample entity object and the three-dimensional positioning association information of the second sample entity object in the sample image may be annotated. The sampled image and the camera intrinsic parameters corresponding to the sampled image are input into the recognition model to obtain a recognition result output by the recognition model, the recognition result including: the first actual entity object, the second actual entity object, the two-dimensional positioning association information of the first actual entity object and the three-dimensional positioning association information of the second actual entity object, the error between the output recognition result and the first sample entity object and the second sample entity object, the two-dimensional positioning association information of the first sample entity object and the three-dimensional positioning association information of the second sample entity object is calculated, and if the error is greater than a preset error threshold, the model parameters are optimized based on back propagation and gradient descent and other technologies until the corresponding error is less than the preset error threshold, and the training of the recognition model is completed.

[0034] In some possible embodiments, a verification data set may also be constructed, the verification data set including a verification image and a verification camera internal parameter, as well as annotated verification images of the first verification entity object and the second verification entity object, and the two-dimensional positioning association information of the first verification entity object and the three-dimensional positioning association information of the second verification entity object. The verification image and the verification camera internal parameter are input into the recognition model at every preset period, and an error comparison is performed based on the output result of the recognition model and the result stored in the verification data set. The reliability of the recognition model is determined based on the error obtained by the comparison. When the error obtained is greater than the preset error value, the model parameters of the recognition model may be adjusted to ensure that the error of the adjusted recognition model is less than the preset error value.

[0035] In one embodiment of the present disclosure, before the target image and the camera intrinsic parameters are input into a pre-trained recognition model, the target image may be pre-processed, that is, the size of the target image is adjusted to a preset uniform size, the target image is color corrected, and it is identified whether the target image is in a preset image format. If it is not in the preset image format, the preset image format is converted, etc.

[0036] In one embodiment of the present disclosure, the target image and the camera intrinsic parameters are taken as input together, and the physical characteristics of the camera can be directly utilized, so that the recognition model can learn during the training process how to optimize the detection of physical objects based on the camera's focal length, distortion and other intrinsic parameters, especially the scale adaptability to physical objects at different distances.

[0037] Step 103, obtaining the two-dimensional positioning association information of the first entity object output by the second detection branch head, the three-dimensional positioning association information of the second entity object output by the third detection branch head, and the entity object type information of the first entity object and the second entity object output by the first detection branch head.

[0038] Among them, the two-dimensional positioning association information may include the first confidence information and bounding box position information of the first entity object, and the three-dimensional positioning association information may include the second confidence information and three-dimensional box position information of the second entity object, wherein the three-dimensional box position information may include orientation, depth distance, center deviation, three-dimensional size, depth value, etc.

[0039] In one embodiment of the present disclosure, Figure 2 As shown, the feature detection layer in the pre-trained recognition model includes three parts, including a first detection branch head, a second detection branch head and a third detection branch head, wherein the second detection branch head outputs two-dimensional positioning association information of the first entity object, the third detection branch head outputs three-dimensional positioning association information of the second entity object, and the first detection branch head outputs entity object type information of the first entity object and the second entity object.

[0040] Among them, the three parts can be a DD3D detection head integrated with YOLOX, etc., and DD3D is divided into the above three detection branch heads. Among them, in the embodiment of the present disclosure, the third detection branch head can be a decoupled detection head, that is, the three-dimensional positioning association information contains multiple three-dimensional positioning information elements, and the multiple three-dimensional positioning information elements include orientation, depth distance, center deviation, three-dimensional size, depth value, etc. In this embodiment, the third detection head may include multiple layers of detection convolution layers, wherein the third detection branch head outputs multiple three-dimensional positioning information elements from the preset detection convolution layer in the multi-layer detection convolution layer, that is, decouples and outputs multiple three-dimensional positioning information elements, which has lower requirements on the chip.

[0041] Among them, DD3D is an end-to-end, single-stage monocular 3D target detection method. It uses point cloud data in training, supervises the generation of depth maps, and shares the feature extraction layer of predicted depth. Point cloud data is not used in actual reasoning. Only target images and camera internal parameters are needed to complete the three-dimensional frame position information. DD3D is a fully convolutional single-stage network extended on FCOS, which can detect three-dimensional frame position information, etc. The recognition model in this embodiment can use FPN as the network extraction layer to extract features and output fused feature maps of multiple downsampling sizes. Three detection heads are applied to each generated feature fusion map for independent prediction tasks. Among them, the second detection branch head predicts the two-dimensional positioning association information of the first entity object, the third detection branch head predicts the three-dimensional positioning association information of the second entity object, and the first detection branch head predicts the entity object type information of the first entity object and the second entity object.

[0042] It should be noted that the preset detection convolution layer for outputting multiple three-dimensional positioning information elements may vary according to different scenarios. In some possible embodiments, refer to Figure 3 After passing through four 3*3 2D convolutional layers, the third detection head will predict multiple three-dimensional positioning information elements (represented as three-dimensional positioning information element 1, three-dimensional positioning information element 2... in the figure), among which the three-dimensional positioning information element can be output between the second and third convolutional layers. In this example, the preset detection convolutional layer is the second convolutional layer.

[0043] In one embodiment of the present disclosure, a first entity object that meets a first preset confidence condition may be determined based on the first confidence information, for example, a first entity object whose first confidence information is higher than a first confidence threshold is determined to be the first entity object that meets the first preset confidence condition, and a second entity object that meets a second preset confidence condition may be determined based on the second confidence information, for example, a second entity object whose first confidence information is higher than a second confidence threshold is determined to be the second entity object that meets the second preset confidence condition. Of course, if the same location contains multiple types of first entity objects that meet the first preset confidence condition, the second confidence information of the multiple types of second entity objects is further judged, and the second entity object corresponding to the highest second confidence information is used as the second entity object at the corresponding location. Similarly, if the same location contains multiple types of second entity objects that meet the second preset confidence condition, the second confidence information of the multiple types of second entity objects is further judged, and the second entity object corresponding to the highest second confidence information is used as the second entity object at the corresponding location. Output the bounding box position information and entity object type information of the first entity object that meets the first preset confidence condition, and output the three-dimensional box position information and entity object type information of the second entity object that meets the second preset confidence condition. In this embodiment, the distance between the vehicle and the corresponding entity object can be further determined based on the bounding box position information and the three-dimensional box position information, and when the distance is less than a preset distance threshold, control the vehicle to brake, etc.

[0044] In one embodiment of the present disclosure, the recognition model also includes multiple feature fusion layers, which are connected in sequence, wherein each feature fusion layer has a different extraction step size for extracting the fusion feature map, thereby achieving different downsampling of the fusion feature map based on different extraction step sizes, thereby ensuring comprehensive recognition of physical objects of various sizes in the near field and far field. In some possible embodiments, the extraction step size of the multiple layers of fusion feature layers connected in sequence can be increased step by step, thereby facilitating the extraction of relevant features of physical objects of various sizes. In the non-first layer of the multi-layer feature fusion layer, the fusion feature map output by the previous feature fusion layer is downsampled according to the corresponding resolution.

[0045] In this embodiment, multiple feature fusion layers are connected to multiple detection layers one by one, and each feature fusion layer inputs the acquired fusion feature map of the corresponding extraction step into the corresponding detection layer. Each detection layer includes a first detection branch head, a second detection branch head, and a third detection branch head. That is to say, for each downsampled fusion feature map, the corresponding entity object recognition result is output. In the actual execution process, it is also possible to apply non-maximum suppression (NMS) and other technologies to remove redundant or low-confidence recognition results and further optimize the output.

[0046] In some possible embodiments, Figure 4 As shown in the figure, when the structure of the fusion feature layer is the fusion feature layer corresponding to the FPN network structure, the three layers P5-P6 can be used as the fusion feature layer, wherein the P5 layer is downsampled by two convolution modules to obtain P6 and P7, and three fusion feature maps with different downsampling rates can be extracted.

[0047] In one embodiment of the present disclosure, the corresponding fusion feature map can be detected by the first detection branch head, the second detection branch head and the third detection branch head in each detection layer, respectively, to obtain the two-dimensional positioning association information of the first entity object output by the second detection branch head in each detection layer, the three-dimensional positioning association information of the second entity object output by the third detection branch head, and the entity object type information of the first entity object and the second entity object output by the first detection branch head.

[0048] Before the fusion feature layer, the recognition model also includes a feature extraction layer, wherein, continue to refer to Figure 4 The feature extraction layer can reuse the feature extraction layer of the network such as FCOS3D, and the feature extraction layer may include C3-C5 layers, etc., wherein, in some possible embodiments, the target image and the camera intrinsic parameters can be input into the first feature extraction layer in the multi-layer feature extraction layer, wherein the first extracted feature map output by the last feature extraction layer in the multi-layer feature extraction layer is input into the first feature fusion layer in the multi-layer feature fusion layer, that is, the multi-layer feature extraction layer synchronously extracts the relevant features of the target image and the camera intrinsic parameters. Among them, the parameters of the first feature extraction layer in the feature extraction layer can be fixed to reduce the memory occupation of the recognition model, etc.

[0049] That is, in this embodiment, YOLOX and DD3D detection heads can be integrated based on the FCOS3D framework, and multiple downsampling rates can be considered at the same time to more comprehensively identify the entity objects contained in the target image. Among them, FCOS3D is the backbone network of the model. The network can effectively extract image features and generate multi-scale fusion feature maps through downsampling to adapt to entity objects of different sizes. Among them, in the recognition model, the YOLOX detection head can be used for 2D bounding box prediction (the YOLOX detection head can be equivalent to the second detection branch head), and its fast and accurate characteristics are suitable for preliminary positioning of entity objects. 3D box prediction is performed using the DD3D detection head (the DD3D detection head can be equivalent to the third detection branch head, etc.), combined with the camera intrinsic parameter information, to output more accurate three-dimensional positioning association information, etc.

[0050] In some possible embodiments, the camera intrinsic parameter features of the camera intrinsic parameters can also be extracted according to a preset neural network, and the target image is input into the first feature extraction layer in the multi-layer feature extraction layer, the second extracted feature map output by the last feature extraction layer is obtained, the second extracted feature map and the camera intrinsic parameter features are fused to obtain a fused feature map, and the fused feature map is input into the first feature fusion layer. That is, in this embodiment, a camera intrinsic parameter feature in the form of a learnable vector is obtained by extracting the camera intrinsic parameter through a small preset neural network, and the second extracted feature map and the camera intrinsic parameter feature are fused before being processed by the fusion feature layer.

[0051] In this technical solution, an efficient and accurate dual-input recognition model is constructed by deeply integrating the architectural advantages of FCOS3D, YOLOX and DD3D. It not only makes full use of the rich image information in the target image, but also cleverly combines the camera intrinsic parameters to enhance the recognition model's understanding and capture of far-field details. Through feature extraction with multiple extraction steps and effective detection head design, it can significantly improve the detection accuracy of physical objects while maintaining high-speed operation, which is especially suitable for the recognition of long-distance physical objects in complex environments.

[0052] In summary, the image recognition processing method of the embodiment of the present disclosure obtains the target image captured by the vehicle camera and obtains the camera intrinsic parameters of the vehicle camera, and then inputs the target image and the camera intrinsic parameters into a pre-trained recognition model, wherein the recognition model includes a first detection branch head, a second detection branch head and a third detection branch head, and obtains the two-dimensional positioning association information of the first entity object output by the second detection branch head, the three-dimensional positioning association information of the second entity object output by the third detection branch head, and the entity object type information of the first entity object and the second entity object output by the first detection branch head. In this technical solution, it is possible to flexibly adapt to images from different camera parameters to identify entity objects contained in the driving environment, and to combine two-dimensional and three-dimensional to jointly identify entity objects, thereby improving the robustness and accuracy of entity object recognition.

[0053] In order to implement the above embodiment, the present disclosure also proposes an image recognition processing device.

[0054] Figure 5 This is a schematic diagram of the structure of an image recognition processing device provided by an embodiment of the present disclosure. The device can be implemented by software and / or hardware and can generally be integrated in an electronic device to perform image recognition processing. Figure 5 As shown, the device includes: a first acquisition module 510, an input module 520 and a second acquisition module 530, wherein:

[0055] A first acquisition module 510 is used to acquire a target image captured by a vehicle camera and acquire camera intrinsic parameters of the vehicle camera;

[0056] An input module 520, used to input the target image and the camera internal parameters into a pre-trained recognition model, wherein the recognition model includes a first detection branch head, a second detection branch head and a third detection branch head;

[0057] The second acquisition module 530 is used to obtain the two-dimensional positioning association information of the first entity object output by the second detection branch head, the three-dimensional positioning association information of the second entity object output by the third detection branch head, and the entity object type information of the first entity object and the second entity object output by the first detection branch head.

[0058] In one embodiment of the present disclosure, the recognition model includes multiple feature fusion layers, which are connected in sequence, wherein each feature fusion layer extracts a fusion feature map with a different extraction step size;

[0059] The multiple feature fusion layers are connected to the multiple detection layers one by one. Each feature fusion layer inputs the acquired fusion feature map of the corresponding extraction step into the corresponding detection layer. Each detection layer includes a first detection branch head, a second detection branch head and a third detection branch head.

[0060] In one embodiment of the present disclosure, a non-first feature fusion layer in the multi-layer feature fusion layer downsamples the fusion feature map output by the previous feature fusion layer according to the corresponding extraction step size.

[0061] In one embodiment of the present disclosure, the second acquisition module 530 is specifically configured to:

[0062] Detecting the corresponding fusion feature map through the first detection branch head, the second detection branch head and the third detection branch head in each detection layer respectively;

[0063] Obtain the two-dimensional positioning association information of the first entity object output by the second detection branch head in each detection layer, the three-dimensional positioning association information of the second entity object output by the third detection branch head, and the entity object type information of the first entity object and the second entity object output by the first detection branch head.

[0064] In one embodiment of the present disclosure, the recognition model further includes multiple feature extraction layers, which are sequentially connected to the input module 520, specifically used for:

[0065] The target image and camera internal parameters are input into the first feature extraction layer in the multi-layer feature extraction layer.

[0066] Wherein, the first extracted feature map output by the last feature extraction layer in the multi-layer feature extraction layer is input into the first feature fusion layer in the multi-layer feature fusion layer; or,

[0067] The camera intrinsic parameter features of the camera intrinsic parameter are extracted according to the preset neural network, and the target image is input into the multi-layer feature extraction layer.

[0068] The second extracted feature map output by the multi-layer feature extraction layer is obtained, the second extracted feature map and the camera intrinsic reference feature are fused to obtain a fused feature map, and the fused feature map is input into the first feature fusion layer.

[0069] In one embodiment of the present disclosure, the two-dimensional positioning association information includes first confidence information and bounding box position information of the first entity object, and the three-dimensional positioning association information includes second confidence information and three-dimensional box position information of the second entity object, and further includes: a processing module for:

[0070] Determining a first entity object that meets a first preset confidence condition according to the first confidence information;

[0071] Determining a second entity object that meets a second preset confidence condition according to the second confidence information;

[0072] The enclosing box position information and entity object type information of the first entity object that meets the first preset reliability condition are output, and the three-dimensional box position information and entity object type information of the second entity object that meets the second preset reliability condition are output.

[0073] In one embodiment of the present disclosure, the three-dimensional positioning association information includes multiple three-dimensional positioning information elements, the third detection branch head includes multiple layers of detection convolution layers, and the multiple three-dimensional positioning information elements are output from a preset detection convolution layer in the multiple layers of detection convolution layers through the third detection branch head.

[0074] The image recognition processing device provided in the embodiments of the present disclosure can execute the image recognition processing method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0075] In order to implement the above embodiments, the present disclosure further proposes a computer program product, including a computer program / instruction, which implements the image recognition processing method in the above embodiments when executed by a processor.

[0076] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure.

[0077] The following specific reference Figure 6, which shows a schematic diagram of the structure of an electronic device 600 suitable for implementing the embodiment of the present disclosure. The electronic device 600 in the embodiment of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0078] like Figure 6 As shown, the electronic device 600 may include a processor (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a memory 608 to a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processor 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0079] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a memory 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0080] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 609, or installed from the memory 608, or installed from the ROM 602. When the computer program is executed by the processor 601, the above-mentioned functions defined in the image recognition processing method of the embodiment of the present disclosure are executed.

[0081] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0082] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0083] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0084] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device:

[0085] The target image captured by the vehicle camera is obtained, and the camera intrinsic parameters of the vehicle camera are obtained, and then the target image and the camera intrinsic parameters are input into a pre-trained recognition model, wherein the recognition model includes a first detection branch head, a second detection branch head and a third detection branch head, and the two-dimensional positioning association information of the first entity object output by the second detection branch head, the three-dimensional positioning association information of the second entity object output by the third detection branch head, and the entity object type information of the first entity object and the second entity object output by the first detection branch head are obtained. In this technical solution, images from different camera parameters can be flexibly adapted to recognize entity objects contained in the driving environment, and entity objects are recognized by combining two-dimensional and three-dimensional methods, thereby improving the robustness and accuracy of entity object recognition.

[0086] The electronic device may be written in one or more programming languages ​​or a combination thereof to write computer program code for performing the operations of the present disclosure, including but not limited to object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0087] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0088] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, constitute a limitation on the unit itself.

[0089] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0090] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0091] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0092] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0093] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. An image recognition processing method, characterized in that: The following steps are involved: Acquire a target image captured by a vehicle camera, and acquire camera intrinsic parameters of the vehicle camera; Inputting the target image and the camera internal parameters into a pre-trained recognition model, wherein the recognition model includes a first detection branch head, a second detection branch head, and a third detection branch head; Acquire the two-dimensional positioning association information of the first entity object output by the second detection branch head, the three-dimensional positioning association information of the second entity object output by the third detection branch head, and the entity object type information of the first entity object and the second entity object output by the first detection branch head.

2. The method according to claim 1, characterized in that The recognition model includes multiple feature fusion layers, which are connected in sequence, wherein the extraction step lengths of the fusion feature maps extracted by each of the feature fusion layers are different; The multiple layers of feature fusion layers are connected to the multiple detection layers one by one, respectively. Each layer of the feature fusion layer inputs the acquired fusion feature map of the corresponding extraction step into the corresponding detection layer. Each of the detection layers includes a first detection branch head, a second detection branch head and a third detection branch head.

3. The method according to claim 2, characterized in that The non-first feature fusion layer in the multi-layer feature fusion layer downsamples the fusion feature map output by the previous feature fusion layer according to the corresponding extraction step size.

4. The method according to claim 2, characterized in that The acquiring of the two-dimensional positioning association information of the first entity object output by the second detection branch head, the three-dimensional positioning association information of the second entity object output by the third detection branch head, and the entity object type information of the first entity object and the second entity object output by the first detection branch head includes: Respectively detecting the corresponding fusion feature map through the first detection branch head, the second detection branch head and the third detection branch head in each of the detection layers; Obtain the two-dimensional positioning association information of the first entity object output by the second detection branch head in each detection layer, the three-dimensional positioning association information of the second entity object output by the third detection branch head, and the entity object type information of the first entity object and the second entity object output by the first detection branch head.

5. The method according to claim 1, characterized in that The recognition model further includes multiple feature extraction layers, which are sequentially connected, and the target image and the camera internal parameters are input into the pre-trained recognition model, including: Input the target image and the camera internal parameters into the first feature extraction layer in the multi-layer feature extraction layer, wherein the first extracted feature map output by the last feature extraction layer in the multi-layer feature extraction layer is input into the first feature fusion layer in the multi-layer feature fusion layer; or, Extracting camera intrinsic parameter features of the camera intrinsic parameter according to a preset neural network, and inputting the target image into the multi-layer feature extraction layer, A second extracted feature map output by the multi-layer feature extraction layer is obtained, the second extracted feature map is fused with the camera intrinsic reference feature to obtain a fused feature map, and the fused feature map is input into the first-layer feature fusion layer.

6. The method according to any one of claims 1 to 5, characterized in that: The two-dimensional positioning association information includes first confidence information and bounding box position information of the first entity object, and the three-dimensional positioning association information includes second confidence information and three-dimensional box position information of the second entity object. The method further includes: Determining a first entity object that meets a first preset confidence condition according to the first confidence information; Determining a second entity object that meets a second preset confidence condition according to the second confidence information; The enclosing box position information and entity object type information of the first entity object that meets the first preset reliability condition are output, and the three-dimensional box position information and entity object type information of the second entity object that meets the second preset reliability condition are output.

7. The method according to any one of claims 1 to 5, characterized in that: The three-dimensional positioning association information includes a plurality of three-dimensional positioning information elements. The third detection branch head includes multiple detection convolution layers, and the plurality of three-dimensional positioning information elements are output from a preset detection convolution layer in the multiple detection convolution layers through the third detection branch head.

8. An image recognition processing device, characterized in that: include: A first acquisition module, used to acquire a target image captured by a vehicle camera and acquire a camera intrinsic parameter of the vehicle camera; An input module, used for inputting the target image and the camera internal parameters into a pre-trained recognition model, wherein the recognition model includes a first detection branch head, a second detection branch head and a third detection branch head; The second acquisition module is used to obtain the two-dimensional positioning association information of the first entity object output by the second detection branch head, the three-dimensional positioning association information of the second entity object output by the third detection branch head, and the entity object type information of the first entity object and the second entity object output by the first detection branch head.

9. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the executable instructions to implement the image recognition processing method described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is used to execute the image recognition processing method described in any one of claims 1 to 7.

Images