Target recognition method based on binocular vision and target recognition model training method

Through image cutting and stitching technology based on binocular vision, the computing power requirements and detection accuracy problems of high-resolution image recognition on the small computing power platform are solved, efficient object detection and model training are achieved, and real-time detection is improved. and accuracy.

CN117934943BActive Publication Date: 2024-11-12元橡科技(北京)有限公司 +1
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202410101228.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-01-24
Publication Date
2024-11-12
Estimated Expiration
2044-01-24

AI Technical Summary

Technical Problem

On the small computing power platform, the target recognition model processing of high-resolution images has problems with excessive computing power requirements, detection accuracy and real-time performance. The existing technology leads to missed detection and misdetection through segmented image processing, and binocular camera parallax The graph cannot accurately obtain the target boundary.

Method used

The object recognition method based on binocular vision is adopted to cut and splice the sampled images through the parallax map, generate the images to be identified, and the target recognition model is trained accordingly to reduce computing power requirements and improve detection efficiency.

Benefits of technology

It effectively reduces the computing power requirement of the target recognition model to process images, improves the real-time and accuracy of object detection, reduces the waste of computing power, and adds negative samples through model training to improve the training effect.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117934943B_ABST
    Figure CN117934943B_ABST
Patent Text Reader

Abstract

The present application discloses a target recognition method based on binocular vision and a target recognition model training method, wherein the target recognition method includes: step 11: obtaining a sampled image and a corresponding disparity map, and based on the disparity in the disparity map, cropping the sampled image to generate an image to be spliced; step 12: splicing the spliced ​​images to generate an image to be recognized; step 13: using the target recognition model, performing target detection on the image to be recognized, generating a target recognition label for the image to be recognized, and mapping the target recognition label to the sampled image. Through the technical solution in the present application, the sampled image is cropped and spliced ​​based on the disparity map, most of the useless image areas in the sampled image can be removed, and the image spliced ​​using such a method can be used for target recognition, which helps to reduce the computing power required by the target recognition model when processing images and reduce the waste of computing power.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of image processing, and in particular to a target recognition method based on binocular vision and a target recognition model training method. Background Art

[0002] In the field of computer vision, images with higher resolutions (such as 1280*720, 1920*1080) often contain richer image information, which is more conducive to the target recognition model to accurately identify the target in the image, but the higher the resolution, the more computing power the platform needs to provide to support it. However, for low-computing platforms (or car platforms that are only allocated a small part of the computing power), it is impossible to directly recognize targets in high-resolution images, which affects the implementation of computer vision technology.

[0003] In the prior art, in order to realize target recognition of high-resolution images on such low-computing platforms, the collected high-resolution images are usually directly segmented into several small images of equal size, and then the segmented small images are input into the platform (model) for recognition, and the final target result is output. It can be understood by those skilled in the art that such a processing method will segment the complete target in the image into different small images, resulting in missed detection and false detection, which in turn affects the accuracy of computer vision target recognition. At the same time, it will also cause a significant decrease in the detection speed, affecting the real-time requirements of the detection.

[0004] As for the binocular camera, since the disparity map it outputs can derive the area with connected outer contours, it is possible to consider cropping and splicing the image to be recognized (RGB image) based on the disparity map to meet the needs of a low-computing power platform, but there is no such related existing technology.

[0005] In addition, since the disparity can only determine the approximate and irregular area of ​​a target and cannot accurately obtain its boundary, the area corresponding to the RGB image to be detected is also irregular, and it is impossible to directly use the existing target detection model for identification. Summary of the invention

[0006] The purpose of this application is to: how to crop and splice large-resolution images to be recognized in order to reduce the computing power required by the target recognition model when processing images, and to train such models accordingly.

[0007] The technical solution of the first aspect of the present application is: a target recognition method based on binocular vision is provided, and the target recognition method includes: step 11: obtaining a sampling image and a corresponding disparity map, and based on the disparity in the disparity map, cropping the sampling image to generate an image to be spliced; step 12: splicing the spliced ​​images to generate an image to be recognized; step 13: using a target recognition model, performing target detection on the image to be recognized, generating a target recognition label for the image to be recognized, and mapping the target recognition label to the sampling image.

[0008] In some embodiments, in step 11, based on the disparity in the disparity map, the sampled image is cropped to generate an image to be spliced, specifically including: determining a potential target area based on the disparity in the disparity map; determining a maximum circumscribed rectangle of the potential target area, and transforming the potential target area to the sampled image by means of coordinate transformation, and recording the area transformed to the sampled image as a preprocessed region of interest; based on the preprocessed region of interest, the sampled image is cropped to generate an image to be spliced.

[0009] In some embodiments, step 12 specifically includes: creating an all-zero image, and selecting any image to be stitched, using a sliding window method, in a left-to-right and top-to-bottom order, positionally matching the selected image to be stitched in the all-zero image; when it is determined that the selected image to be stitched is completely located in the all-zero image, recording the position of the selected image to be stitched in the all-zero image as the area to be stitched; calculating the convolution value of the pixel value of the selected image to be stitched and the pixel value in the area to be stitched; determining whether the convolution value is 0, if so, stitching the selected image to be stitched to the area to be stitched, generating an image to be identified, and selecting the next image to be stitched, re-position matching, if not, re-position matching, and selecting the next area to be stitched.

[0010] In some embodiments, the target recognition method further includes: when generating the image to be stitched, recording the first position information of each image to be stitched in the sampling image; when generating the image to be identified, recording the second position information of each image to be stitched in the image to be identified; in step 13, mapping the target recognition label to the sampling image, specifically including: according to the first position information and the second position information, mapping the target recognition label of the image to be identified to the sampling image by coordinate mapping.

[0011] The technical solution of the second aspect of the present application is: a target recognition model training method is provided, and the training method includes: step 21: based on the label information of the target frame marked in the sample image, determine the preset ratio, and expand the size of the target frame based on the preset ratio to generate a cropping frame; step 22: based on the cropping frame, crop the sample image, and record the cropped image as a candidate image; step 23: splicing multiple candidate images to generate a training image, and based on the training image, train the target recognition model.

[0012] In some embodiments, step 23 specifically includes: step 231: determining at least one candidate target frame corresponding to the candidate image based on the cropping frame and coordinate calculation method, and generating a candidate target frame sequence based on the candidate target frame, wherein the candidate target frame is a target frame located within the cropping frame and has an overlapping area with the cropping frame; step 232: splicing multiple candidate images to generate a training image; step 233: splicing the candidate target frame sequence based on the splicing order of multiple candidate images to generate a training target frame sequence corresponding to the training image; step 234: training the target recognition model based on the training image and the training target frame sequence.

[0013] In some embodiments, step 232 specifically includes: selecting any candidate image, determining the target category label corresponding to the candidate image, wherein the target category label is determined by the label information of the target box corresponding to the candidate image; determining the scaling ratio based on the target category label according to preset rules; scaling the candidate image based on the scaling ratio; and splicing multiple scaled candidate images to generate a training image.

[0014] In some embodiments, step 231 specifically includes: calculating the relative position coordinates of any candidate target frame and the size of the candidate target frame; wherein the relative position coordinates are the coordinates of the center point of the candidate target frame in the candidate image; based on the scaling ratio, transforming the relative position coordinates and scaling the size of the candidate target frame; generating a candidate target frame sequence based on the target category label corresponding to the candidate target frame, the transformed relative position coordinates and the scaled candidate target frame size.

[0015] In some embodiments, in step 23, multiple candidate images are spliced ​​to generate a training image, specifically including: creating a full-value image based on a preset size, and selecting any candidate image and recording it as the current candidate image, wherein the pixel value in the full-value image is 0; using a sliding window method, the current candidate image is position-matched in the full-value image in a left-to-right and top-to-bottom order; when it is determined that the current candidate image is completely located in the full-value image, the position of the current candidate image in the full-value image is recorded as the area to be spliced; calculating the convolution value of the pixel value of the current candidate image and the pixel value in the area to be spliced; determining whether the convolution value is 0, if so, splicing the current candidate image to the area to be spliced ​​to generate a training image, and selecting the next candidate image and re-recording it as the current candidate image, if not, re-position matching is performed and the area to be spliced ​​is selected.

[0016] In some embodiments, step 21 further includes: when it is determined that the coordinates of the cropping frame exceed the boundary coordinates of the sample image, updating the coordinates of the cropping frame based on the boundary coordinates.

[0017] The beneficial effects of this application are:

[0018] The technical solution in the present application crops and splices the image to be identified (sampled image) based on the disparity map to obtain the image to be identified for target recognition. Most of the useless image areas in the sampled image can be removed, which helps to reduce the computing power required for the target recognition model when processing the image, realizes target detection under high resolution and low computing power, and improves the real-time performance of target detection.

[0019] In order to enable the target recognition model to recognize the images to be recognized obtained by the above-mentioned cropping and splicing, the present application also proposes a corresponding model training method. Taking into account that the target frame of the sample image is usually a positive sample and its size is smaller than the image size captured by disparity, the target frame is expanded and then the sample image is cropped and spliced ​​to obtain the corresponding training image. On the one hand, the size of the target area in the disparity map and the target area in the re-cropped sample image can be basically corresponding; on the other hand, the negative samples in the images used in training are increased, thereby improving the training effect and training efficiency of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The advantages of the above and / or additional aspects of the present application will become apparent and easily understood in the description of the embodiments in conjunction with the following drawings, in which:

[0021] Figure 1 is a schematic flow chart of a binocular vision-based target recognition method according to an embodiment of the present application;

[0022] Figure 2 is a schematic diagram of a target recognition workflow according to an embodiment of the present application;

[0023] Figure 3 is a schematic diagram of a disparity map according to an embodiment of the present application;

[0024] Figure 4 is a target recognition model training method according to an embodiment of the present application;

[0025] Figure 5 is a schematic diagram of a labeled sample image according to an embodiment of the present application;

[0026] Figure 6 is a schematic diagram of a candidate image according to an embodiment of the present application;

[0027] Figure 7 is a schematic diagram of candidate target frame positions according to an embodiment of the present application;

[0028] Figure 8 is a schematic diagram of a candidate target frame sequence displayed in a candidate image according to an embodiment of the present application;

[0029] Fig. 9 is a schematic diagram of a training image according to an embodiment of the present application;

[0030] Fig.10 is a schematic diagram of the coordinate position of the clipping frame according to an embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0032] In the following description, many specific details are elaborated to facilitate a full understanding of the present application. However, the present application may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present application is not limited to the specific embodiments disclosed below.

[0033] Embodiment 1:

[0034] like Figure 1 and Figure 2 As shown, this embodiment provides a target recognition method based on binocular vision, and the target recognition method includes:

[0035] Step 11: Obtain a sampled image and a corresponding disparity map, and based on the disparity in the disparity map, crop the sampled image to generate an image to be stitched;

[0036] Step 12: stitch the images to be stitched to generate the image to be recognized;

[0037] Step 13: Use the target recognition model to perform target detection on the image to be recognized, generate a target recognition label for the image to be recognized, and map the target recognition label to the sampled image.

[0038] Specifically, for a binocular camera, the disparity map corresponding to the left image (right image) can be calculated based on the disparity calculation principle through the acquired left and right images (RGB images), and then the left image (right image) and the disparity map can be combined. The category information of the target in front of the vehicle can be obtained by using the left image (right image), and the position information of the corresponding target can be obtained by using the disparity map, so as to realize the detection / recognition of obstacles in front of the vehicle.

[0039] However, in existing image target detection / recognition methods, all pixels in the image need to be processed indiscriminately and the category label of each pixel needs to be predicted. This processing method not only increases the number of tasks that the model (platform) needs to handle, but also wastes a lot of computing power on useless image areas, affecting the efficiency of target recognition.

[0040] like Figure 3 As shown in the figure, in three-dimensional space, different targets in the disparity map are not adjacent in space (such as the areas with different colors in the figure), but for a single target, its outer contour is continuous in space. Therefore, based on the spatial (continuous) characteristics of the target in the disparity map, the disparity data can be processed using a segmentation algorithm (such as clustering, connected domain analysis), thereby segmenting the areas of different targets in the disparity map.

[0041] Therefore, the target area obtained from the disparity map can be combined with the corresponding sampled image (left image) to crop the sampled image to obtain the image to be spliced. This type of image to be spliced ​​is the image area containing various types of targets in the sampled image. Then, the images to be spliced ​​are spliced ​​by image splicing to obtain the image to be recognized, so as to be input into the target recognition model for image target recognition.

[0042] By using the above-mentioned image cropping and stitching method, most of the useless image areas in the sampled image can be removed, avoiding the model from wasting computing power on processing useless image areas, which helps to achieve target detection with high resolution and low computing power, and improves the real-time performance of target detection.

[0043] In addition, for the images to be stitched, those skilled in the art can understand that, since their positions in the sampled images correspond one-to-one to their positions in the images to be identified, the target identification labels in the images to be identified can be mapped to the sampled images by mapping.

[0044] By comparing the target recognition method in this embodiment with the conventional target recognition method, the target recognition results of the two are basically the same, and the target recognition method in this embodiment requires about 50% of the computing power required by the conventional target recognition method. Therefore, the method of cutting and splicing the image to be recognized (sampled image) based on the disparity map proposed in this embodiment helps to reduce the computing power required when the target recognition model processes the image and reduce the waste of computing power.

[0045] In some embodiments, in step 11, based on the disparity in the disparity map, the sampled image is clipped to generate the image to be stitched, which specifically includes:

[0046] Based on the disparity in the disparity map, determine the potential target area, that is, the area segmented based on the disparity value;

[0047] Determine the maximum circumscribed rectangle of the potential target area, and transform the potential target area to the sampling image by means of coordinate transformation, and record the area transformed to the sampling image as the preprocessing region of interest, and record the position information of the preprocessing region of interest, such as the coordinates of the upper left corner, length and width, etc.;

[0048] Based on the preprocessed region of interest, the sampled image is cropped to generate the image to be stitched.

[0049] Specifically, for the multiple images to be stitched obtained through the above process, they can be scaled according to a preset ratio and then stitched. When stitching, the images can be sorted in the order of the coordinates of the upper left corners of the images to be stitched, and then stitched in order; the multiple images to be stitched can also be sorted according to length / width, and then stitched in order. Therefore, in the spliced ​​image to be recognized, the resolution details of the original sampled image can be retained to the greatest extent, and then combined with the deep learning network for target recognition, a target recognition result close to the resolution details of the original sampled image can be obtained.

[0050] In some embodiments, step 12 specifically includes:

[0051] Create an all-zero image, select any image to be stitched, and use a sliding window to match the position of the selected image to be stitched in the all-zero image from left to right and from top to bottom;

[0052] When it is determined that the selected image to be stitched is completely located in the all-zero image, the position of the selected image to be stitched in the all-zero image is recorded as the area to be stitched;

[0053] Calculate the convolution value of the pixel value of the selected image to be stitched and the pixel value in the area to be stitched;

[0054] Determine whether the convolution value is 0. If so, stitch the selected image to be stitched to the area to be stitched to generate an image to be recognized, select the next image to be stitched, and re-match the position. If not, re-match the position and select the next area to be stitched.

[0055] Specifically, based on a preset size (such as 160*160), an all-zero image with all pixel values ​​of 0 is created, and the images to be spliced ​​are selected in sequence for position matching.

[0056] Taking the current i-th image to be stitched as an example, the position matching process of multiple images to be stitched is described.

[0057] Each sampled image is used as a unit to generate the image to be recognized, and the i-th image to be spliced ​​is used as the current image to be spliced. Its upper left corner is taken as the reference, and a sliding window is used to match its position in the all-zero image from left to right and from top to bottom.

[0058] Since the sizes of each image to be stitched are not consistent, in order to improve the efficiency of image stitching, the convolution idea is introduced. The specific stitching process is as follows:

[0059] If the current image to be stitched is completely located in the all-zero image, then the convolution of the pixel value of the current image to be stitched and the pixel value at the same position of the all-zero image is calculated;

[0060] However, it is understood by those skilled in the art that the pixel values ​​in the current image to be stitched are not all 0. Therefore, if the value of the above convolution result is 0, it means that the pixel values ​​of the all-zero image at that position are all 0, and the position is taken as the area to be stitched, the current image to be stitched is stitched at the position, the pixel value of the all-zero image at the position is replaced, the position coordinates of the center point of the current image to be stitched in the all-zero image are recorded, and the next image to be stitched is selected, and the above process is re-executed;

[0061] If the value of the convolution result is not 0, it means that a certain image to be spliced ​​has been placed (spliced) at this position of the all-zero image. At this time, it is necessary to use a sliding window method, from left to right and from top to bottom, to determine the position of the current image to be spliced ​​in the all-zero image again, and recalculate the convolution value of the pixel;

[0062] If the right image of the current image to be stitched cannot be completely located in the all-zero image, it means that the current image to be stitched cannot be stitched in the current row of the all-zero image, and the above process should be repeated starting from the left side of the next row of the all-zero image.

[0063] Similarly, if the image below the current image to be stitched cannot be completely located in the all-zero image, it means that the current image to be stitched cannot be stitched in this all-zero image, that is, the current all-zero image has been filled with the previous image to be stitched, or the remaining space in the current all-zero image is insufficient to place the current image to be stitched. At this time, it is necessary to recreate an all-zero image, and then stitch the current image to be stitched in the newly created all-zero image.

[0064] It should be noted that in order to further reduce the demand for computing power, the images to be stitched may be scaled, the images may be reduced in proportion, and then the scaled images to be stitched may be stitched.

[0065] In some embodiments, in order to ensure the accuracy of the target identification tag mapping, a mapping table may be used. The target identification method further includes:

[0066] When generating the images to be stitched, the generated images to be stitched are numbered, and the first position information of each image to be stitched in the sampled image is recorded in the mapping table in sequence; wherein the first position information may be the coordinates of the center point of the image to be stitched in the sampled image, or may be the coordinates of the upper left corner of the image to be stitched in the sampled image;

[0067] When generating the image to be identified, the images to be spliced ​​are selected in order, and the second position information of each image to be spliced ​​in the image to be identified is recorded in the mapping table in turn; wherein the second position information should correspond to the first position information, and is one of the center point coordinates or the upper left corner coordinates;

[0068] In step 13, the target identification label is mapped to the sampled image, which specifically includes:

[0069] According to the first position information and the second position information, the target recognition label of the image to be recognized is mapped to the sampled image by means of coordinate mapping.

[0070] Embodiment 2:

[0071] When the sampled image is intercepted / cropped for the target to be identified based on the disparity map, part of the background area around the target to be identified will be included, increasing the image range of the target to be identified. Therefore, the sample image needs to be processed so that it can provide training images with a size basically consistent with the image intercepted by disparity during the training process, so as to ensure that the model can accurately identify the target with less computing power requirements.

[0072] In addition, for images captured by parallax, the added background part is usually a negative sample. Since the target box in the sample image usually only selects the target to be identified, if the image is captured only based on the target box and the sample image is reconstructed, the reconstructed sample image only includes positive samples. Using such reconstructed sample images to train the target recognition model will seriously affect the accuracy of target recognition.

[0073] It is understood by those skilled in the art that the resolution (size) of the images used in the model training stage and the recognition stage should be consistent. Therefore, the sample images used for training need to be cropped and spliced ​​in the same way. However, considering that the existing sample images usually do not have corresponding disparity maps, Figure 4 As shown, this embodiment provides a target recognition model training method, the method comprising:

[0074] Step 21: determining a preset ratio based on the label information of the target frame marked in the sample image, and expanding the size of the target frame based on the preset ratio to generate a cropping frame;

[0075] Step 22: based on the cropping frame, crop the sample image, and record the cropped image as a candidate image;

[0076] Step 23: Multiple candidate images are stitched together to generate training images, and the target recognition model is trained based on the training images.

[0077] Specifically, when detecting / identifying obstacles in front of a vehicle based on binocular vision technology, it is necessary to combine the disparity map with the sampled image (the left image or the right image obtained by the binocular camera), use the sampled image to obtain the category information of the target in front of the vehicle, and use the disparity map to obtain the position information of the corresponding target. Therefore, the detection / identification of obstacles in front of the vehicle can be achieved by combining the deep learning network with the binocular vision technology.

[0078] It is understood by those skilled in the art that deep learning networks usually require a large number of sample images for training. For sample images, each sample image is annotated with data, such as Figure 5 As shown in the figure, the targets in the image will be marked one by one according to the set target recognition type, including the target category, center point coordinates, and target box size (width, height), so as to train the deep learning network.

[0079] Among them, for the target (target box) category, it is usually reflected in the form of a label, which can be set manually according to the actual situation. For example, the target category of pedestrians is set to 0, the target category of cars is set to 1, the target category of buses is set to 2, the target category of trucks is set to 3, etc.

[0080] Due to the characteristics of the current disparity technology, in the process of obtaining the target area based on the disparity map, part of the background area around the target to be identified will be divided into the target area, that is, the range of the target area determined in the disparity map is larger than the range of the target area (target box) determined in the sampled image.

[0081] In addition, if image cropping, rotation, scaling, splicing and other operations are performed only based on the target box in the sample image, the spliced ​​image will only contain positive samples. After training with such images, other targets in the background area will be identified as similar positive samples, affecting the recall rate of the trained target recognition model.

[0082] Therefore, it is necessary to expand each target frame in the sample image and determine the cropping frame. On the one hand, the target area in the disparity map and the target area in the re-cropped sample image should have basically corresponding sizes; on the other hand, the negative samples in the images used for training should be increased.

[0083] When expanding any target frame in any sample image, the target frame can be expanded along the length direction and the width direction respectively, wherein the preset ratios of the expansion in the length direction and the width direction can be the same or different, and the preset ratio can be set manually, such as several multiples of the length and width of the original target frame (such as 1.6 times, 2 times).

[0084] Therefore, the sample image can be re-cropped according to the generated cropping frame to obtain the candidate image. Figure 5 Take the truck in the figure as an example, the corresponding candidate image is Figure 6 shown.

[0085] In some embodiments, in order to enrich the samples in the model training process, the cropped candidate images are also automatically annotated and the corresponding label information is obtained. The above step 23 specifically includes:

[0086] Step 231: determining at least one candidate target frame corresponding to the candidate image based on the cropping frame and the coordinate calculation method, and generating a candidate target frame sequence based on the candidate target frame, wherein the candidate target frame is a target frame located within the cropping frame and having an overlapping area with the cropping frame;

[0087] Step 232: splicing multiple candidate images to generate a training image;

[0088] Step 233: based on the stitching order of the multiple candidate images, stitching the candidate target frame sequence to generate a training target frame sequence corresponding to the training image;

[0089] Step 234: Train the target recognition model based on the training images and the training target frame sequence.

[0090] Specifically, for the candidate image that is expanded and re-cropped, if Figure 6 As shown in the figure, not only negative samples (such as trees and buildings) but also some positive samples (such as vehicles on both sides of the road) are added to the candidate images. Therefore, by automatically labeling such positive samples, the number of samples in the model training process can be enriched.

[0091] When automatically labeling other targets in the candidate image, based on the cropping frame range corresponding to the candidate image, the coordinate traversal method can be used to traverse each target frame contained in the corresponding sample image to determine whether the upper left corner coordinates and lower right corner coordinates of each target frame are within the cropping frame range, such as Figure 7 shown.

[0092] For the candidate image 402 determined by the target frame 401, the positional relationships between the remaining target frames and the candidate image 402 include the following three types:

[0093] 1. The target frame is completely outside the candidate image 402 (corresponding cropping frame), such as the target frame 403;

[0094] 2. The target frame is partially located within the candidate image 402 (corresponding cropping frame), such as the target frame 404;

[0095] 3. The target box is completely located within the candidate image 402 (the corresponding cropping box), such as the target box 405.

[0096] Therefore, based on the coordinates of the upper left corner and the lower right corner of any target box, it is possible to determine whether the target box overlaps with the candidate image (the corresponding cropping box) by comparing the coordinates:

[0097] As for the target frame 403, it is sufficient to discard the target frame;

[0098] For the target frame 404, a rectangular coordinate system can be established with the coordinates of the upper left corner of the candidate image as the origin, and the relative position coordinates of the center point of the overlapping part in the candidate image and the size (length, width) of the overlapping part can be obtained by coordinate calculation. The overlapping part is determined as the candidate target frame, and the target category label of the original target frame is inherited;

[0099] For the target frame 405 , the relative position coordinates of the center point of the target frame 405 in the candidate image are also calculated, and the candidate target frame inherits the size (length, width) and target category label of the original target frame 405 .

[0100] Therefore, the overlapping part of target frame 404 and target frame 405 are candidate target frames of the candidate image. Combined with the label information of the candidate target frames, i.e., the relative position coordinates of the center point of the candidate target frame in the candidate image, the size of the candidate target frame, and the target category label inherited from the target frame by the candidate target frame, the corresponding candidate target frame sequence can be obtained.

[0101] When calculating the relative position coordinates of the center point of the candidate target frame in the candidate image, the upper left corner coordinates of the candidate image 402 can be used as the origin to establish a rectangular coordinate system, and the relative coordinates of the target frame 401, the target frame 405 and the overlapping part of the target frame 404 can be obtained, and then the relative position coordinates of the center point can be obtained. In summary, the generated candidate target frame sequence is displayed in the candidate image as follows Figure 8 shown.

[0102] In some embodiments, before stitching to generate training images, the candidate images are also scaled so that the training images contain more candidate images to improve the training efficiency of the model. The above step 232 specifically includes:

[0103] Select any candidate image and determine the target category label corresponding to the candidate image, wherein the target category label is determined by the label information of the target frame corresponding to the candidate image;

[0104] Based on the target category label, the scaling ratio is determined according to the preset rules;

[0105] Based on the scaling ratio, scaling the candidate image;

[0106] Multiple scaled candidate images are stitched together to generate training images.

[0107] Specifically, any candidate image corresponds to a target frame in the sample image, and each target frame in the sample image corresponds to a target category label. Considering that the sizes of the candidate images (including the targets) are not the same, for example, the candidate image corresponding to a truck is larger than the candidate image corresponding to a car, different scaling ratios should be set for candidate images of different categories.

[0108] Therefore, based on the target category label of the set target box, such as pedestrian, car, truck, bus, etc., the scaling ratio is determined according to the preset rules. The specific preset rules are as follows:

[0109] 1) Shortest side protection principle: During the object scaling process, the shortest side of the object frame (image) cannot be less than a certain pixel, such as 20px, 30px or other values;

[0110] 2) For the pedestrian target box, if the shortest side of the target box min(w_b,h_b) is greater than or equal to 21px, the scaling ratio is determined according to the ratio 21 / min(w_b,h_b); if the shortest side is less than 21px, no scaling is performed;

[0111] 3) For the car target box, if the shortest side min(w_b,h_b) of the target box is greater than or equal to 29px, the scaling ratio is determined according to the ratio of 29 / min(w_b,h_b); if the shortest side is less than 29px, no scaling is performed;

[0112] 4) For large vehicle (such as trucks, buses) target boxes, if the shortest side of the target box min(w_b,h_b) is greater than or equal to 36px, the scaling ratio is determined according to the ratio of 36 / min(w_b,h_b); if the shortest side is less than 36px, no scaling is performed.

[0113] It should be noted that the specific preset rules for determining the scaling ratio can be manually set according to actual conditions.

[0114] In some embodiments, step 231 specifically includes:

[0115] Calculate the relative position coordinates of any candidate target frame and the size of the candidate target frame; wherein the relative position coordinates are the coordinates of the center point of the candidate target frame in the candidate image;

[0116] Based on the scaling ratio, the relative position coordinates are transformed and the size of the candidate target frame is scaled, wherein the scaling ratio is determined by the target category label corresponding to the candidate image according to a preset rule;

[0117] A candidate target frame sequence is generated based on the target category label corresponding to the candidate target frame, the transformed relative position coordinates, and the scaled candidate target frame size.

[0118] Specifically, as shown in Table 1, for the i-th target frame in any sample image, the i-th candidate image can be cropped after expansion based on a preset ratio, and the i-th candidate image contains at least one candidate target frame, that is, a target frame that is completely located in the cropping frame (candidate image) and has an overlapping area with the cropping frame.

[0119] Table 1

[0120]

[0121] Therefore, in order to ensure the consistency of label information in the candidate target frame before and after scaling the candidate image, the relative position coordinates and size (length, width) of the candidate image before scaling should be transformed based on the scaling ratio to obtain the relative position coordinates and size of the scaled candidate image, and generate the corresponding candidate target frame sequence, in which the target category label of the candidate target frame remains unchanged.

[0122] In this embodiment, the label information in the candidate target frame includes: the relative position coordinates of the candidate target frame, the size of the candidate target frame, and the target category label inherited by the candidate target frame from the target frame.

[0123] In some embodiments, in step 23, multiple candidate images are stitched together to generate a training image, which specifically includes:

[0124] Based on the preset size, create a full-value image, and select any candidate image as the current candidate image, where the pixel value in the full-value image is 0;

[0125] Using a sliding window, the current candidate image is matched in the full-value image in order from left to right and from top to bottom;

[0126] When it is determined that the current candidate image is completely located in the full-value image, the position of the current candidate image in the full-value image is recorded as the area to be stitched;

[0127] Calculate the convolution value of the pixel value of the current candidate image and the pixel value in the area to be stitched;

[0128] Determine whether the convolution value is 0. If so, splice the current candidate image to the area to be spliced, generate a training image, and select the next candidate image and re-record it as the current candidate image. If not, re-perform position matching and select the area to be spliced.

[0129] Specifically, when stitching multiple candidate images, the resolution (preset size) of the training image can be set based on low computing power requirements, a full-value image with all pixel values ​​​​being 0 is created, and the positions of multiple candidate images are matched in turn.

[0130] Taking the current i-th candidate image as an example, the position matching process of multiple candidate images is described.

[0131] During matching, each sample image is used as a unit to generate a training image, and the i-th candidate image is used as the current candidate image. Its upper left corner is taken as the reference, and a sliding window is used to match its position in the full-value image from left to right and from top to bottom.

[0132] Since the sizes of each candidate image are not consistent, in order to improve the efficiency of image stitching, the convolution idea is introduced. The specific stitching process is as follows:

[0133] If the current candidate image is completely within the full-value image, the convolution of the pixel value of the current candidate image and the pixel value at the same position of the full-value image is calculated;

[0134] However, those skilled in the art can understand that the pixel values ​​in the current candidate image are not all 0. Therefore, if the value of the above convolution result is 0, it means that the pixel values ​​of the full-value image at this position are all 0, and this position is taken as the area to be spliced, the current candidate image is spliced ​​at this position (the area to be spliced), the pixel value of the full-value image at this position is replaced, the position coordinates of the center point of the current candidate image in the full-value image are recorded, the label information of each candidate target frame in the current candidate image is inherited, and the next candidate image is selected to re-execute the above process;

[0135] If the value of the convolution result is not 0, it means that a candidate image has been placed (stitched) at this position of the full-value image. At this time, it is necessary to use a sliding window method, from left to right and from top to bottom, to re-determine the position of the current candidate image in the full-value image and recalculate the convolution value of the pixel;

[0136] If the right image of the current candidate image cannot be completely located in the full-value image, it means that the current candidate image cannot be spliced ​​into the current row in the full-value image, and the above process should be repeated starting from the left side of the next row.

[0137] If the image below the current candidate image cannot be completely located in the full-value image, it means that the current candidate image cannot be spliced ​​into this full-value image, that is, the current full-value image has been filled with the previous candidate image, or the remaining space in the current full-value image is insufficient to place the current candidate image. At this time, it is necessary to recreate a full-value image, and then splice the current candidate image into the newly created full-value image.

[0138] It should be noted that in order to make the generated stitched image more beautiful and avoid leaving large areas of blank space, the pixel values ​​in the full-value image can be set to any value in [10,245], such as 144. At this time, an index map of the same size (160*160) needs to be added. The pixel value in the index map is 0. The above convolution process is performed using the index map and the candidate image. According to the convolution result of the candidate image and the index map, the candidate image is stitched at the corresponding position of the full-value image. The specific process will not be repeated. At this time, the stitched training image is as follows: Fig. 9 shown.

[0139] In some embodiments, step 21 further includes: when it is determined that the coordinates of the cropping frame exceed the boundary coordinates of the sample image, updating the coordinates of the cropping frame based on the boundary coordinates.

[0140] Specifically, Fig.10 As shown, taking 2 times as an example, the range of the expanded cropping frame is twice the size of the original target frame, and the coordinates of the cropping frame (four fixed-point coordinates) need to be determined after an "out-of-bounds" judgment based on the size of the coordinate values, that is, to determine whether the coordinates of the cropping frame exceed the boundary coordinates of the sample image.

[0141] In the "out-of-bounds" judgment process, the upper left corner coordinates and the lower right corner coordinates of the target frame 202 can be calculated respectively according to the set expansion multiple, and the calculation results can be compared with the boundary coordinates (coordinate range) of the sample image 201. The dotted line part 204 of the expanded cropping frame 203 is set as the out-of-bounds part, which needs to be cropped, that is, the coordinates (upper left corner coordinates, lower right corner coordinates) of the cropping frame 203 are updated according to the boundary coordinates of the sample image 201, and the length and width of the cropping frame 203 and the corresponding center point coordinates can be obtained according to the updated coordinates of the cropping frame 203.

[0142] It should be noted that the coordinates of the upper left corner and the lower right corner of the target frame / cropping frame can be calculated based on the coordinates of the center point and the length and width.

[0143] The technical solution in this embodiment is to expand the target frame in the sample image with high resolution (such as 1980*1080) based on a preset ratio, and then crop the sample image to obtain a smaller candidate image, and then determine the scaling ratio again according to the target category label corresponding to the candidate image, and scale the candidate image, so as to facilitate splicing multiple reduced candidate images into one or more full-value images to generate a training image with a small resolution (such as 106*160). In addition, during the above-mentioned scaling, splicing, etc., the center point coordinates, the size of the target frame, and the target category label are processed synchronously, so that the generated training image can be directly used for the training of the target recognition model, which helps to increase the number of samples and improve the model training effect while reducing the model computing power requirements.

[0144] So far, various embodiments of the present application have been described in detail. In order to avoid obscuring the concept of the present application, some details known in the art are not described. Based on the above description, those skilled in the art can fully understand how to implement the technical solution disclosed herein.

[0145] Although some specific embodiments of the present application have been described in detail through examples, those skilled in the art should understand that the above examples are only for illustration rather than for limiting the scope of the present application.

[0146] The steps in this application can be adjusted in order, combined, and deleted according to actual needs.

[0147] Although the present application is disclosed in detail with reference to the accompanying drawings, it should be understood that these descriptions are merely exemplary and are not intended to limit the application of the present application. The scope of protection of the present application is defined by the appended claims and may include various modifications, alterations and equivalents made to the invention without departing from the scope and spirit of the present application.

Claims

1. A target recognition method based on binocular vision, characterized in that: The target recognition method comprises: Step 11: obtaining a sampled image and a corresponding disparity map, and based on the disparity in the disparity map, cutting the sampled image to generate an image to be stitched; wherein: Based on the disparity in the disparity map, the sampled image is clipped to generate an image to be stitched, specifically including: Determining a potential target area based on the disparity in the disparity map; Determine the maximum circumscribed rectangle of the potential target area, and transform the potential target area to the sample image by means of coordinate transformation, and record the area transformed to the sample image as a preprocessed region of interest; Based on the preprocessed region of interest, the sampled image is clipped to generate the image to be stitched; Step 12: stitching the images to be stitched to generate an image to be recognized; Step 13: Using the target recognition model, perform target detection on the image to be recognized, generate a target recognition label for the image to be recognized, and map the target recognition label to the sampled image.

2. The binocular vision-based target recognition method according to claim 1, characterized in that: The step 12 specifically includes: Create an all-zero image, select any image to be stitched, and use a sliding window method to match the position of the selected image to be stitched in the all-zero image in a left-to-right and top-to-bottom order; When it is determined that the selected image to be stitched is completely located in the all-zero image, the position of the selected image to be stitched in the all-zero image is recorded as the area to be stitched; Calculating the convolution value of the pixel value of the selected image to be stitched and the pixel value in the area to be stitched; Determine whether the convolution value is 0, If yes, the selected image to be stitched is stitched to the area to be stitched to generate the image to be identified, and the next image to be stitched is selected to perform position matching again. If not, re-match the position and select the next area to be spliced.

3. The binocular vision-based target recognition method according to any one of claims 1 to 2, characterized in that: The target recognition method further comprises: When generating the images to be stitched, recording first position information of each of the images to be stitched in the sampled image; When generating the image to be identified, recording the second position information of each of the images to be spliced ​​in the image to be identified; In step 13, mapping the target identification label to the sample image specifically includes: According to the first position information and the second position information, the target recognition label of the image to be recognized is mapped to the sampling image by means of coordinate mapping.

4. A target recognition model training method, characterized in that: The training method is applicable to training the target recognition model in the target recognition method based on binocular vision according to any one of claims 1 to 3, and the training method comprises: Step 21: determining a preset ratio based on label information of the target frame marked in the sample image, and expanding the size of the target frame based on the preset ratio to generate a cropping frame; Step 22: based on the cropping frame, cropping the sample image, and recording the cropped image as a candidate image; Step 23: splicing the plurality of candidate images to generate a training image, and training the target recognition model based on the training image.

5. The target recognition model training method according to claim 4, characterized in that: The step 23 specifically includes: Step 231: determining at least one candidate target frame corresponding to the candidate image based on the cropping frame and the coordinate calculation method, and generating a candidate target frame sequence based on the candidate target frame, wherein the candidate target frame is a target frame located within the cropping frame and having an overlapping area with the cropping frame; Step 232: splicing the plurality of candidate images to generate a training image; Step 233: splicing the candidate target frame sequence based on the splicing order of the plurality of candidate images to generate a training target frame sequence corresponding to the training image; Step 234: training the target recognition model based on the training image and the training target frame sequence.

6. The target recognition model training method according to claim 5, characterized in that: The step 232 specifically includes: Select any candidate image and determine the target category label corresponding to the candidate image, wherein the target category label is determined by the label information of the target frame corresponding to the candidate image; Based on the target category label, according to a preset rule, determining a scaling ratio; Based on the scaling ratio, scaling the candidate image; The multiple scaled candidate images are spliced ​​to generate the training image.

7. The target recognition model training method according to claim 6, characterized in that: The step 231 specifically includes: Calculate the relative position coordinates of any candidate target frame and the size of the candidate target frame; wherein the relative position coordinates are the coordinates of the center point of the candidate target frame in the candidate image; Based on the scaling ratio, transform the relative position coordinates and scale the size of the candidate target frame; The candidate target frame sequence is generated based on the target category label corresponding to the candidate target frame, the transformed relative position coordinates, and the scaled candidate target frame size.

8. The target recognition model training method according to claim 4, characterized in that: In step 23, the step of splicing the plurality of candidate images to generate a training image specifically includes: Based on the preset size, a full-value image is created, and any candidate image is selected and recorded as the current candidate image, wherein the pixel value in the full-value image is 0; Using a sliding window method, the current candidate image is positionally matched in the full-value image in a sequence from left to right and from top to bottom; When it is determined that the current candidate image is completely located in the full-value image, the position of the current candidate image in the full-value image is recorded as the area to be stitched; Calculating the convolution value of the pixel value of the current candidate image and the pixel value in the area to be stitched; Determine whether the convolution value is 0, If yes, the current candidate image is spliced ​​to the area to be spliced ​​to generate the training image, and the next candidate image is selected and recorded as the current candidate image again. If not, re-match the positions and select the area to be spliced.

9. The target recognition model training method according to claim 4, characterized in that: The step 21 further includes: When it is determined that the coordinates of the cropping frame exceed the boundary coordinates of the sample image, the coordinates of the cropping frame are updated based on the boundary coordinates.

Citation Information

Cited By

  • Target identification method based on multi-view vision and target identification model training method

    CN120147817A