Binocular vision-based target detection method, device, electronic equipment, and medium

Abstract

The embodiments of the present application provide a method, device, electronic device and medium for target detection based on binocular vision, which relates to the field of computer vision detection technology. The method includes: collecting a binocular image of a scene to be identified, obtaining at least one first perspective detection frame and at least one second perspective detection frame based on the binocular image, obtaining a plurality of calibration feature points in the first perspective detection frame and the second perspective detection frame respectively, and each calibration feature point is fixed in the relative position of the first perspective detection frame and the second perspective detection frame respectively according to the calibration feature points, and obtaining the target detection result of the scene to be identified based on the first feature and the second feature. The embodiment of the present application extracts the calibration feature points of the detection frame and performs target detection through the calibration feature points, which can reduce the large amount of computing power consumed in extracting features from the image and improve the efficiency of target detection.

Description

Technical Field

[0001] The present application relates to the field of computer vision detection technology. Specifically, the present application relates to a binocular vision-based target detection method, device, electronic device, computer-readable storage medium and computer program product. Background Art

[0002] With the continuous development of computer image processing technology, binocular vision detection has become a relatively common target detection method. In subway environments, due to the unique characteristics of the equipment and scenes, foreign object detection and ranging algorithms need to be implemented using portable devices. However, portable devices only provide limited computing power. In scenarios where multiple algorithms are integrated, it is necessary to ensure real-time performance and reduce computing power consumption.

[0003] Existing binocular vision methods use feature extraction algorithms to extract features from binocular images, obtain overall image features, and perform feature matching to obtain detection results. This consumes a lot of computing power, resulting in low overall efficiency. Summary of the Invention

[0004] The embodiments of the present application provide a target detection method, device, electronic device, computer-readable storage medium and computer program product based on binocular vision, aiming to solve the technical problems of high computing power consumption and low efficiency of target detection in existing binocular vision detection methods.

[0005] In a first aspect, a binocular vision-based target detection method is provided, the method comprising:

[0006] Collect binocular images of the scene to be identified;

[0007] Based on the binocular image, obtaining at least one first-view detection frame and at least one second-view detection frame;

[0008] Acquire multiple calibration feature points in the first viewing angle detection frame and the second viewing angle detection frame respectively, wherein the relative positions of the calibration feature points in the first viewing angle detection frame and the second viewing angle detection frame are fixed;

[0009] Determine the first feature and the second feature according to the calibrated feature points, where the first feature is the feature corresponding to the first viewing angle detection frame, and the second feature is the feature corresponding to the second viewing angle detection frame;

[0010] Based on the first feature and the second feature, an object detection result of the scene to be identified is obtained.

[0011] Optionally, the binocular image includes a first-perspective image and a second-perspective image;

[0012] Obtaining at least one first-view detection frame and at least one second-view detection frame based on the binocular image, including:

[0013] Performing preliminary target detection on the first-perspective image in the binocular image to obtain at least one first-perspective detection frame;

[0014] Perform preliminary target detection on the second-view image in the binocular image to obtain at least one second-view detection frame.

[0015] Optionally, determining the first feature and the second feature respectively according to the calibrated feature points includes:

[0016] Establishing pixel coordinate systems on the first-view image and the second-view image respectively;

[0017] Determine the coordinates of multiple calibrated feature points of the first viewing angle detection frame in the pixel coordinate system to obtain a first feature;

[0018] The coordinates of a plurality of calibrated feature points of the second viewing angle detection frame in the pixel coordinate system are determined to obtain a second feature.

[0019] Optionally, obtaining an object detection result of the scene to be identified based on the first feature and the second feature includes:

[0020] Obtaining a first feature descriptor based on the coordinates of each calibrated feature point in the first feature;

[0021] Obtaining a second feature descriptor based on the coordinates of each calibrated feature point in the second feature;

[0022] The first feature descriptor and the second feature descriptor are compared to obtain an object detection result of the scene to be identified.

[0023] Optionally, obtaining an object detection result of the scene to be identified based on the first feature and the second feature includes:

[0024] Matching the first feature with the second feature to obtain a matching result;

[0025] Based on the matching results, the target detection results of the scene to be identified are obtained.

[0026] Optionally, the first feature is matched with the second feature to obtain a matching result, including:

[0027] For each first feature, matching each second feature with the first feature one by one;

[0028] If the match is successful, the first feature is associated with the second feature to obtain an associated feature pair.

[0029] Optionally, based on the matching results, a target detection result of the scene to be identified is obtained, including:

[0030] Determine the first-view detection frame and the second-view detection frame corresponding to the associated feature pair respectively;

[0031] Determine the corresponding detection target based on the first perspective detection frame and the second perspective detection frame corresponding to the associated features;

[0032] Based on the detected target, the target detection result of the scene to be identified is obtained.

[0033] Optionally, based on the detected target, a target detection result of the scene to be identified is obtained, including:

[0034] Determine a first center point of a first-view detection frame corresponding to the associated feature pair;

[0035] Determine a second center point of a second perspective detection frame corresponding to the associated feature pair;

[0036] Determining depth information of the detection target based on the first center point and the second center point;

[0037] Based on the depth information, the target detection result of the scene to be identified is obtained.

[0038] In a second aspect, a binocular vision-based target detection device is provided, the device comprising:

[0039] An acquisition module, used to acquire binocular images of the scene to be identified;

[0040] A detection frame determination module, configured to obtain at least one first-view detection frame and at least one second-view detection frame based on the binocular image;

[0041] A feature point determination module is used to obtain a plurality of calibration feature points in the first viewing angle detection frame and the second viewing angle detection frame, respectively, wherein the relative position of each calibration feature point in the first viewing angle detection frame and the second viewing angle detection frame is fixed;

[0042] A feature acquisition module is used to determine a first feature and a second feature based on the calibrated feature points, wherein the first feature is a feature corresponding to the first viewing angle detection frame, and the second feature is a feature corresponding to the second viewing angle detection frame;

[0043] The result acquisition module is used to obtain the target detection result of the scene to be identified based on the first feature and the second feature.

[0044] According to a third aspect, an electronic device is provided, comprising:

[0045] A memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of any method in the first aspect of the present application.

[0046] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the binocular vision-based target detection method shown in any one of the first aspects of the present application is implemented.

[0047] In a fifth aspect, a computer program product is provided, comprising a computer program, which implements the steps of any one of the methods in the first aspect of the present application when the computer program is executed by a processor.

[0048] The beneficial effects of the technical solution provided by the embodiments of the present application are:

[0049] The binocular vision-based target detection method provided by the present application obtains target detection frames of the two perspectives in the binocular image, determines calibration feature points at predetermined positions corresponding to each target detection frame, obtains features of each detection frame based on the calibration feature points, and thus obtains target detection results of the scene to be identified based on features corresponding to the first perspective detection frame and the second perspective detection frame. The calibrated feature points of the detection frames of different perspectives are used to identify targets in the image, and accurate target detection results can be calculated in a timely manner with a small amount of data. The method is suitable for portable devices with small computing power in special scenarios such as subways, and effectively improves the efficiency of target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments of the present application.

[0051] Figure 1 A schematic diagram of an application scenario of a binocular vision-based target detection method provided in an embodiment of the present application;

[0052] Figure 2 A flowchart of a binocular vision-based target detection method provided in an embodiment of the present application;

[0053] Figure 3 A schematic diagram of calibrating feature points in a binocular vision-based target detection method provided in an embodiment of the present application;

[0054] Figure 4 A schematic diagram of a process for preliminary target detection in a binocular vision-based target detection method provided in an embodiment of the present application;

[0055] Figure 5 A schematic diagram of a pixel coordinate system in a binocular vision-based target detection method provided in an embodiment of the present application;

[0056] Figure 6 A schematic diagram of a process for matching a first feature with a second feature in a binocular vision-based target detection method provided in an embodiment of the present application;

[0057] Figure 7 A schematic diagram of depth information in a binocular vision-based target detection method provided in an embodiment of the present application;

[0058] Figure 8 A flowchart illustrating an example of a binocular vision-based target detection method provided in an embodiment of the present application;

[0059] Figure 9 A schematic structural diagram of a binocular vision-based target detection device provided in an embodiment of the present application;

[0060] Figure 10 A schematic structural diagram of an electronic device applicable to a binocular vision-based target detection method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0061] The following describes the embodiments of the present application in conjunction with the accompanying drawings. It should be understood that the embodiments described below in conjunction with the accompanying drawings are exemplary descriptions for explaining the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions of the embodiments of the present application.

[0062] Those skilled in the art will understand that, unless otherwise stated, the singular forms "a", "an", "said", and "the" used herein may also include plural forms. It should be further understood that the terms "including" and "comprising" used in the embodiments of the present application mean that the corresponding features can be implemented as the presented features, information, data, steps, operations, elements, and / or components, but do not exclude implementation as other features, information, data, steps, operations, elements, components, and / or combinations thereof supported by the present technical field. It should be understood that when we say that an element is "connected" or "coupled" to another element, the element can be directly connected or coupled to the other element, or it can refer to the element and the other element establishing a connection relationship through an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The terms "or", "and / or", "including at least one of the following", etc. used in this application may be interpreted as inclusive, or mean any one or any combination.

[0063] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or portion of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal. It can be implemented in whole or in part using software, hardware (such as processing circuits or memory), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the functionality of the module or unit.

[0064] In the specific implementation of this application, any data related to an object, such as data involved in the object's use of an application, is required. When the embodiments of this application are applied to specific products or technologies, the permission or consent of the object must be obtained, and the collection, use, and processing of the relevant data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. In other words, if any of the above-mentioned data related to an object is involved in the embodiments of this application, such data must be obtained with the authorization and consent of the object and in compliance with the relevant laws, regulations, and standards of the relevant countries and regions.

[0065] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0066] First, the technical terms involved in this application are introduced and explained:

[0067] Binocular image: refers to a pair of images captured simultaneously by two cameras from different perspectives. The two cameras are usually arranged at a certain distance and angle to simulate the visual system of the human eyes. It is mainly used for stereo vision and 3D reconstruction.

[0068] In existing technologies, target detection in subway environments requires the ability to detect foreign objects and quickly identify the type and distance of foreign objects. Traditional railway locomotives have long-standing production lines, and the realization of new intelligent requirements cannot be directly invested in production line improvements without long-term market verification. Therefore, portable intelligent equipment has a wide range of application scenarios in subway environments. However, railway operations have strict electrical standards. Using LiDAR for identification and ranging functions requires locomotives to provide high-power power to embedded devices. This results in some LiDAR products being limited in size and power supply and unable to be better applied to embedded devices. Therefore, most of them use optical sensors.

[0069] Current optical ranging methods primarily include monocular depth estimation, binocular ranging, and structured light or time-of-flight (ToF) sensor ranging. Monocular depth estimation relies on deep learning models or geometric information to infer depth information from a single image. Its accuracy depends on the quality of model training and some prior information, which leads to poor performance in scenarios where the presence of foreign objects is unknown and computing power is limited. Structured light and ToF offer millimeter-level accuracy, but their operating range is typically short, making them unsuitable for use in subway environments and their cost is high. Therefore, we use binocular ranging for foreign object detection and distance calculation.

[0070] In industrial applications, binocular vision methods often use the entire image to calculate dense disparity maps, which requires large amounts of computation and consumes computing power. In subway environments, due to the particularity of the equipment and scenes, relatively more tasks need to be processed by portable devices, while embedded devices can only provide limited computing power support. In the scenario of multi-algorithm integration, existing solutions have higher requirements for computing power, which is not conducive to multi-algorithm integration and has low efficiency.

[0071] During target recognition, binocular feature matching is required. Traditional feature matching methods take the entire image as input, extract pre-defined feature points, and then perform matching based on left and right object descriptors. However, in multi-target detection or foreign object detection, objects appearing in the same frame for both left and right eyes do not occupy the entire image pixel space. Consequently, foreign object detection is not frequent or widespread in rail transit or special operations, a particularly significant issue in subway environments.

[0072] The binocular vision-based target detection method, device, electronic device, computer-readable storage medium and computer program product provided in this application are intended to solve at least one of the above technical problems in the prior art.

[0073] In response to at least one of the above-mentioned technical problems or areas that need improvement in the related art, the present application proposes a target detection method, device, electronic device, computer-readable storage medium and computer program product based on binocular vision. The target detection method based on binocular vision provided by this solution obtains the target detection frames of the two perspectives in the binocular image, and determines the calibration feature points corresponding to each target detection frame. The features of each detection frame are obtained based on the calibration feature points, and the target detection results of the scene to be identified are obtained based on the features corresponding to the first perspective detection frame and the second perspective detection frame. The calibrated feature points of the detection frames of different perspectives are used to identify the target in the image. Accurate target detection results can be calculated in a timely manner with a small amount of data. It is suitable for portable devices with small computing power in special scenarios such as subways, and effectively improves the efficiency of target detection.

[0074] Furthermore, the pixel coordinate system corresponding to the first-perspective detection frame is determined based on the first-perspective image, and the pixel coordinate system corresponding to the second-perspective detection frame is determined based on the second-perspective image, so as to obtain the coordinate information of the calibrated feature points corresponding to each detection frame and the corresponding feature descriptors. By comparing the feature descriptors corresponding to the calibrated feature points of two different perspectives, a more accurate target recognition result can be obtained.

[0075] In addition, the first feature and the second feature corresponding to the first perspective detection frame and the second perspective detection frame are determined according to the feature calibration points, and matching is performed based on the features to obtain a successfully matched associated feature pair, thereby determining the detection target and depth information corresponding to the detection frame, which can quickly and accurately realize the recognition and ranging of the detection target.

[0076] The following describes several exemplary embodiments to illustrate the technical solutions of the embodiments of the present application and the technical effects produced by the technical solutions of the present application. It should be noted that the following embodiments can refer to, draw on, or combine with each other, and the same terms, similar features, and similar implementation steps in different embodiments will not be repeated.

[0077] Figure 1 A schematic diagram of an application scenario of a binocular vision-based target detection method provided in an embodiment of the present application, wherein the application environment may include subway equipment 100, and the subway equipment 100 includes a first-perspective camera device 200 and a second-perspective camera device 300.

[0078] Specifically, the subway equipment 100 collects binocular images of the scene to be identified through the first-view camera device 200 and the second-view camera device 300. Based on the binocular images, the subway equipment 100 obtains at least one first-view detection frame and at least one second-view detection frame, and obtains multiple calibration feature points in the first-view detection frame and the second-view detection frame respectively. The relative position of each calibration feature point in the first-view detection frame and the second-view detection frame is fixed. According to the calibration feature points, the first feature corresponding to the first-view detection frame and the second feature corresponding to the second-view detection frame are determined respectively. Based on the first feature and the second feature, the target detection result of the scene to be identified is obtained.

[0079] The above application scenario is only an example and does not limit the application scenario of the binocular vision-based target detection method of this application.

[0080] In some possible implementations, taking the execution subject as a subway device as an example, the embodiment of the present application provides a target detection method based on binocular vision, such as Figure 2 As shown, the following steps may be included:

[0081] S210: Acquire a binocular image of the scene to be recognized.

[0082] Among them, the scenes to be identified may include subway scenes, traffic scenes, autonomous driving scenes, industrial manufacturing scenes, drone scenes, and smart home appliance scenes.

[0083] Specifically, two cameras are used to capture the scene to be identified, and images with two different perspectives, namely binocular images, are obtained. During the acquisition, a common trigger signal can be used to ensure that the two cameras capture images at the same time, thereby ensuring synchronous acquisition of binocular images.

[0084] During the specific implementation process, binocular image data can be collected through a visible light sensor, and the synchronization mechanism of the robot operating system ROS2 can be used to ensure the time synchronization of the binocular images. In order to ensure image quality and reduce calibration errors, the two cameras can be of the same model and set on the same horizontal line to reduce vertical parallax and simplify subsequent processing. The distance between the two cameras depends on the required three-dimensional measurement range and accuracy.

[0085] S220 : Obtain at least one first-viewing angle detection frame and at least one second-viewing angle detection frame based on the binocular image.

[0086] The binocular image includes images from two perspectives, a first perspective image and a second perspective image, and each detection frame corresponds to a detection object.

[0087] Specifically, a first-perspective image and a second-perspective image are obtained from the binocular image, and preliminary target detection is performed on the first-perspective image and the second-perspective image respectively to obtain at least one first-perspective detection frame corresponding to the first perspective and at least one second-perspective detection frame corresponding to the second perspective.

[0088] During the specific implementation process, the image can be input into the detection frame prediction model to obtain the corresponding detection frame, the first-perspective image can be input into the detection frame prediction model to obtain the corresponding first-perspective detection frame, and the second-perspective image can be input into the detection frame prediction model to obtain the corresponding second-perspective detection frame. The detection frame prediction model can be trained based on the following method: obtain multiple image samples, each image sample is marked with a corresponding detection frame label, and the image samples are input into the initial prediction model in sequence to obtain a predicted detection frame. According to the predicted detection frame and the detection frame label, the parameters of the initial prediction model are updated until the predetermined end condition is reached to obtain a trained detection frame prediction model.

[0089] S230 , obtaining a plurality of calibration feature points in the first viewing angle detection frame and the second viewing angle detection frame respectively.

[0090] Among them, the calibrated feature points include multiple feature points within the first perspective detection frame and the second perspective detection frame. The relative positions of each calibrated feature point in the first perspective detection frame and the second perspective detection frame are fixed, and can include points on the detection frame line and points within the detection frame.

[0091] Specifically, relevant information of the first perspective detection frame and relevant information of the second perspective detection frame are obtained, and based on the preset calibration point positions and relevant information corresponding to each detection frame, corresponding calibration feature points are determined at the preset positions of the detection frames, wherein the relevant information may include the length and width information of the detection frame and the coordinate information in the corresponding coordinate system of the image, etc.

[0092] In the specific implementation process, Figure 3 As shown in the figure, the calibration feature points can be set to nine feature points within the detection frame, including four outer points and five inner points. The four outer points can include the four corner points of the detection frame, and the five inner points are based on the center point of the detection frame, and the center point is moved up, down, left and right by a quarter of the height and width of the detection frame to obtain another four inner points, thus forming the five inner points. Since the area framed by the detection frame is usually not completely consistent with the actual shape and boundary of the target object, and usually contains mixed information of the target and background, resulting in a sudden change in texture and brightness, this texture mutation has obvious characteristics, and the four corner points of the detection frame need to be taken into consideration. The five inner points are mainly concentrated in the main part of the detection frame. The points within a certain range of the center of the detection frame usually contain relatively rich texture information, which is usually more concentrated on the target body, with less interference and conducive to feature matching, and high robustness. Combining the high contrast of the four outer points and the rich texture information of the five inner points can achieve more accurate feature matching goals, which can accelerate the feature matching process while maintaining good accuracy.

[0093] In the specific implementation process, the position of the feature points corresponding to the calibrated feature points can be adjusted as needed. It only needs to include the points on the detection frame and the points inside the detection, taking into account both edge and internal features to ensure the accuracy of feature extraction. For example, Figure 3 As shown, the outer four points can include four corner points, and the inner five points can include the center point of the detection frame and four points translated a certain distance up, down, left, and right, and the positions of the five inner points are in the shape of a cross; the outer four points can also be the midpoints of the four sides of the detection frame, and the inner five points can include the center point of the detection frame and four points moved a certain distance toward the four corners, and the positions of the five inner points are in the shape of an oblique cross. The specific feature point positions can be set based on scene characteristics, image characteristics or recognition requirements.

[0094] In the specific implementation process, the number of feature points corresponding to the calibrated feature points can be selected and set according to factors such as actual detection accuracy and computing power limitations. Generally, it is at least five. The performance achieved by the nine feature points set above is better, which can ensure the accuracy of feature recognition and reduce redundancy. For example, if the number of feature points is five, it can include four corner points on the detection box and one center point in the detection box. If the number of feature points is eight, it can include four corner points on the detection box and four points in the detection box. If the number of feature points is ten, it can include four corner points on the detection box and the center points of the two long sides, as well as four points in the center.

[0095] S240: Determine the first feature and the second feature respectively according to the calibrated feature points.

[0096] Among them, the first feature is the feature corresponding to the first viewing angle detection frame, and the second feature is the feature corresponding to the second viewing angle detection frame.

[0097] Specifically, based on the calibrated feature points corresponding to the first perspective detection frame, the pixel points within a certain range around each calibrated feature point are determined, the features of the pixel points within the range are analyzed, and the first feature corresponding to the first perspective detection frame is determined. Based on the calibrated feature points corresponding to the second perspective detection frame, the second feature of the second perspective detection frame is obtained in the same way, so as to achieve stereo matching of detection frames of different perspectives based on the features corresponding to the detection frames, wherein the features of the pixel points include brightness, texture, pixel intensity, etc.

[0098] During the specific implementation process, for each calibrated feature point, the reference pixels around the feature point are determined based on the preset range box, and the brightness of each reference pixel is compared to obtain the corresponding feature identifier until the feature identifier of the corresponding feature point is obtained. Based on the feature identifier of each feature point corresponding to the detection frame, the feature of the corresponding detection frame is determined. For example, based on the feature identifier of each feature point in the detection frame of the first perspective, the first feature is obtained, and based on the feature identifier of each feature point in the detection frame of the second perspective, the second feature is obtained. By comparing the brightness of the pixel points, the detection frame features can be more accurately reflected, which is conducive to the subsequent matching of the detection frames of the two perspectives and more accurately determining whether they are the same object.

[0099] S250: Obtaining a target detection result of the scene to be identified based on the first feature and the second feature.

[0100] The scene to be identified includes at least one detection target, and each detection target includes a corresponding detection result.

[0101] Specifically, based on the first feature corresponding to each first-perspective detection frame and the second feature corresponding to each second-perspective detection frame, the features corresponding to the two perspectives are matched, the detection object in the detection frame is identified, and the successfully matched detection object is used as the detection target, thereby obtaining the detection result of the corresponding detection target, until the detection results of all matching detection objects in the scene to be identified are obtained, and the target detection result of the scene to be identified is obtained.

[0102] In the specific implementation process, preliminary target detection is used to represent each detection object in the binocular image in the form of a detection frame, and then stereo matching is performed on the detection frame features of different perspectives (also called left and right eyes) to determine the detection targets that can be matched. Among them, the stereo matching algorithm relies on the result output of the preliminary target detection, and on this basis correctly matches the same target in the binocular image, calculates the target depth value, and finally outputs it together with the detection result. Compared with the existing recognition and ranging scheme, this application obtains the detection frame information of the target in the image through the target detection algorithm, constructs a stereo matching basis based on the position information, matches the same target in the same scene in the binocular image, and calculates the distance between the target and the subject. It requires fewer sensors, is more conducive to the use of embedded devices, requires less computing power support, and can also maintain the accuracy of recognition and ranging.

[0103] In some possible implementations, obtaining at least one first-view detection frame and at least one second-view detection frame based on the binocular image in the above steps includes:

[0104] Performing preliminary target detection on the first-perspective image in the binocular image to obtain at least one first-perspective detection frame;

[0105] Perform preliminary target detection on the second-view image in the binocular image to obtain at least one second-view detection frame.

[0106] The binocular image includes a first-perspective image and a second-perspective image.

[0107] Specifically, preliminary target detection is performed on the first-perspective image in the binocular image to obtain at least one first-perspective detection frame, and preliminary target detection is performed on the second-perspective image in the binocular image to obtain at least one second-perspective detection frame. When obtaining the detection frame, the coordinates of the upper left corner of the detection frame in the pixel coordinate system of the corresponding image and the number of pixels of the length and width of the detection frame can also be obtained, so as to subsequently perform stereo matching on the features of the detection frames of different perspectives.

[0108] In the specific implementation process, Figure 4As shown in the figure, in the preliminary target detection part, the binocular image can include the left and right eyes. The target detection algorithm YOLOX can be used to detect multiple targets in the left and right images respectively. The binocular image is input into the feature extraction network Backbone to extract feature information in the image. The extracted features of different scales are then efficiently fused through the feature fusion network Neck to generate a multi-scale feature map suitable for target detection. The final detection results are then generated through the detection head network Head, including target category prediction, boundary detection box regression, and target confidence estimation. In addition, based on the convolutional neural network, the backbone network CSPDarkNet can be constructed to achieve efficient image feature extraction. By constructing the target detection network PAFPN structure, efficient feature fusion is achieved. Finally, a decoupled detection head is used to complete the output of multi-target detection results.

[0109] In some possible implementations, determining the first feature and the second feature based on the calibrated feature points in the above steps includes:

[0110] Establishing pixel coordinate systems on the first-view image and the second-view image respectively;

[0111] Determine the coordinates of multiple calibrated feature points of the first viewing angle detection frame in the pixel coordinate system to obtain a first feature;

[0112] The coordinates of a plurality of calibrated feature points of the second viewing angle detection frame in the pixel coordinate system are determined to obtain a second feature.

[0113] The unit of the pixel coordinate system is the size of one image pixel.

[0114] Specifically, a first pixel coordinate system is established on the first perspective image, and the coordinates of multiple calibrated feature points of the first perspective detection frame in the pixel coordinate system are determined to obtain a first feature. A second pixel coordinate system is established on the second perspective image, and the coordinates of multiple calibrated feature points of the second perspective detection frame in the pixel coordinate system are determined to obtain a second feature. The number of pixels of the first perspective image and the second perspective image can be the same.

[0115] In the specific implementation process, Figure 5 As shown, the pixel coordinate system can take the upper left corner of the image as the coordinate axis origin O, with the horizontal rightward direction as the positive direction of the horizontal coordinate (i.e., u shown in the figure), and the vertical downward direction as the positive direction of the vertical coordinate (i.e., v shown in the figure). It can also take the center point of the image as the origin, with the horizontal rightward direction as the positive direction of the horizontal coordinate, and the horizontal upward direction as the positive direction of the vertical coordinate. The coordinate system can also be established with the lower left corner, upper right corner, or lower right corner of the image as the origin. The specific settings are made according to actual needs.

[0116] In some possible implementations, obtaining the target detection result of the scene to be identified based on the first feature and the second feature in the above steps includes:

[0117] Obtaining a first feature descriptor based on the coordinates of each calibrated feature point in the first feature;

[0118] Obtaining a second feature descriptor based on the coordinates of each calibrated feature point in the second feature;

[0119] The first feature descriptor and the second feature descriptor are compared to obtain an object detection result of the scene to be identified.

[0120] Specifically, for each calibrated feature point, based on the coordinates of the feature point in the pixel coordinate system, the pixel points within a preset range adjacent to the feature point are determined, the brightness of any two reference pixel points in the same range is compared, and the corresponding identification value is determined based on the brightness, thereby determining the corresponding feature descriptor. For the first feature, the first feature descriptor is obtained using the above method, and for the second feature, the second feature descriptor is obtained. The first feature descriptor and the second feature descriptor are matched. If the match is successful, the detection target is determined based on the corresponding detection frame, thereby determining the depth information of the detection target and obtaining the target detection result of the scene to be identified.

[0121] In the specific implementation process, the binary robust independent basic feature descriptor BRIEF can be used to calculate the descriptor corresponding to the feature point. A square window can be defined around the feature point. For example, a square window with a side length of 30 pixels is used as an example. Two pixels are randomly selected in the window as pixel pairs for brightness comparison to obtain the corresponding feature descriptor. Let the neighborhood area of ​​each point be ,exist Randomly select two pixels within and , for each pixel pair , compare their brightness values and , generates a binary value as follows:

[0122]

[0123] in, is the binary value of the pixel pair. Repeat the above process to compare n pairs of pixels. Taking n as 256 as an example, a binary descriptor D with a length of n is generated, as shown below:

[0124]

[0125] In the specific implementation process, taking the number of feature points as 9 as an example, each detection box will generate 9 256-dimensional binary descriptors. By comparing the similarity between descriptors, it can be determined whether two detection boxes identify the same object. The similarity between descriptors can be measured by the Hamming distance, as shown in the following formula:

[0126]

[0127] in, is the Hamming distance between the feature point descriptor D1 and the feature point descriptor D2, n is the number of bits of the descriptor, i is the index of each bit of the descriptor, for The binary value of the i-th bit in , yes The binary value of the i-th bit in , Represents a bitwise XOR operation, which describes the number of different bits between the two descriptors. When the Hamming distance between the descriptors of all feature points is 0, that is, all are the same, it means that the first-view detection frame matches the second-view detection frame.

[0128] In some possible implementations, obtaining the target detection result of the scene to be identified based on the first feature and the second feature in the above steps includes:

[0129] Matching the first feature with the second feature to obtain a matching result;

[0130] Based on the matching results, the target detection results of the scene to be identified are obtained.

[0131] The first feature and the second feature may include corresponding brightness, texture, shape, etc.

[0132] Specifically, based on the calibrated feature points, the first feature corresponding to the first perspective detection frame and the second feature corresponding to the second perspective detection frame are determined respectively, the first feature is matched with the second feature, and the matching result is determined. If the match is successful, it is considered that the detection objects in the corresponding detection frames are the same, and the detection object is used as the detection target, and the target detection result corresponding to the detection target is obtained. By matching the first feature with the second feature, it can be determined whether the two detection frames identify the same object.

[0133] In the specific implementation process, the above-mentioned matching of the first feature with the second feature and determining the matching result may include: comparing the brightness value corresponding to the first feature with the brightness value corresponding to the second feature, if the difference in the brightness values ​​is less than a preset brightness threshold, the match is considered successful, otherwise it fails; matching the texture type corresponding to the first feature with the texture type corresponding to the second feature, if the texture types are the same, the match is considered successful, otherwise it fails; comparing the shape corresponding to the first feature with the shape corresponding to the second feature, a pre-trained shape comparison model can be used for comparison to obtain the similarity, if the shape similarity is greater than the preset threshold, the match is considered successful, otherwise it fails.

[0134] In some possible implementations, matching the first feature with the second feature in the above step to obtain a matching result includes:

[0135] For each first feature, matching each second feature with the first feature one by one;

[0136] If the match is successful, the first feature is associated with the second feature to obtain an associated feature pair.

[0137] Here, for each second feature, each first feature may be matched with the second feature one by one.

[0138] Specifically, for each first feature, each second feature is matched with the first feature one by one. If the match is successful, the first feature is associated with the corresponding second feature to obtain an associated feature pair. The detection object in the corresponding detection frame of the associated feature pair is the detection target. The detection results such as the depth information of the detection target can be determined based on the associated feature pair.

[0139] In the specific implementation process, the detection frame of the binocular image is determined, and the detection frames of the same target are paired based on the matching logic. For example, the first-view detection frame is the left-eye detection frame, and the second-view detection frame is the right-eye detection frame. The matching process is as follows: Figure 6 As shown, the meanings of the symbols shown in the figure are as shown in the following table:

[0140] Table 1:

[0141]

[0142] Specifically, the two view detection frames are stereo matched to identify the same detection target. The process includes: setting i to 0, j to 1, and judging whether i+1 exceeds the total number of left view detection frames (as shown in the figure). If i+1 exceeds the number of left eye detection frames, the stereo matching is terminated; if it does not exceed the number of left eye detection frames, the value of i is set to i+1 and the corresponding left eye detection frame is determined, and the right eye detection frame corresponding to the identifier j is determined, and the feature points of the left eye detection frame (i.e., L shown in the figure) and the right eye detection frame (i.e., R shown in the figure) are matched to determine whether the left and right eye detection frames are the same target. If the match is successful (i.e., L shown in the figure), the stereo matching is terminated. i =R j ), then calculate the depth information of the detection target corresponding to the detection frame (that is, d(L i , R j )), update the right eye detection frame set and delete the matched right eye detection frame (that is, R=RR shown in the figure j ), and set j to 1, set the value of the left eye detection frame identifier i to i+1, and match the next left eye detection frame; if the match is not successful, set the value of j to j+1 until all right eye detection frames are matched.

[0143] In the specific implementation process, if the left and right eye detection frames are the same target, it is judged whether the incremented identifier i+1 does not exceed the number of left eye detection frames. If it does not exceed the number of left eye detection frames, the left eye detection frame information of the corresponding identifier is obtained and a new round of judgment is performed. If it exceeds the number, the stereo matching is terminated. If the condition that the left and right eye detection frames are the same target is not met, it is judged whether the incremented identifier j+1 exceeds the number of right eye detection frames. If it does not exceed, the right eye detection frame of the corresponding identifier is obtained and continues to match with the left eye detection frame. If the number exceeds, the value of j is set to 1, and it is judged whether the current i+1 exceeds the number of left eye detection frames. If it does not exceed, the value of i is set to i+1, and a new round of matching judgment is started. Otherwise, the stereo matching is terminated.

[0144] In some possible implementations, obtaining the target detection result of the scene to be identified based on the matching result in the above steps includes:

[0145] Determine the first-view detection frame and the second-view detection frame corresponding to the associated feature pair respectively;

[0146] Determine the corresponding detection target based on the first perspective detection frame and the second perspective detection frame corresponding to the associated features;

[0147] Based on the detected target, the target detection result of the scene to be identified is obtained.

[0148] Specifically, the first perspective detection frame and the second perspective detection frame corresponding to the associated feature pair are determined respectively, and the corresponding detection target is identified based on the first perspective detection frame and the second perspective detection frame corresponding to the associated feature pair. The features of the detection target can be analyzed to determine the size, position and category of the detection target, and the target detection result of the scene to be identified is obtained.

[0149] During the specific implementation process, the detection frame information corresponding to the associated feature pair is determined, and the detection frame information is analyzed to obtain the information of the detection target. A deep learning model, such as a category prediction model, can be used to input the information of the first-view detection frame and the second-view detection frame into the category prediction model to obtain the predicted category of the corresponding detection target. For subway scenes, the category of the detection target may include the distance to the preceding vehicle, foreign objects, animals, and other objects. The above-mentioned category prediction model can be trained based on the following method: obtain multiple detection frame sample pairs, each detection frame sample pair is marked with a corresponding category label, and input the detection frame sample pairs into the initial prediction model in sequence to obtain the predicted category. According to the predicted category and category label, the parameters of the initial prediction model are updated until the predetermined end condition is met to obtain a trained category prediction model.

[0150] In some possible implementations, obtaining a target detection result of a scene to be identified based on the detected target in the above steps includes:

[0151] Determine a first center point of a first-view detection frame corresponding to the associated feature pair;

[0152] Determine a second center point of a second perspective detection frame corresponding to the associated feature pair;

[0153] Determining depth information of the detection target based on the first center point and the second center point;

[0154] Based on the depth information, the target detection result of the scene to be identified is obtained.

[0155] Specifically, the depth information of the corresponding detection target is calculated based on the detection frames of the two perspectives, and the first center point of the first perspective detection frame and the second center point of the second perspective detection frame are obtained. Based on the distance between the first center point and the second center point, the distance of the binocular camera device, and the focal length of the binocular camera device, the depth information of the detection target is calculated, and all successfully matched detection targets and corresponding depth information in the binocular image of the scene to be identified are obtained to obtain the target detection result of the scene to be identified.

[0156] In the specific implementation process, for the successfully matched associated feature pairs, the center point corresponding to the detection frame is determined, and the distance to the detection target is calculated using the binocular camera imaging principle based on the center point. Figure 7 As shown, P represents the detection target, and are the optical centers of the binocular cameras, b represents the baseline distance between the cameras, and They represent the projection points of the detection target on the imaging plane, that is, the first center point and the second center point, respectively. f represents the focal length (the distance between the imaging plane and the optical center). Here, the horizontal right direction is the main direction of the x-axis. It is easy to know that and Similarly, the depth value calculation formula is as follows:

[0157]

[0158]

[0159] in, is the depth value, f is the focal length, b is the distance of the optical center of the binocular camera device, is the horizontal coordinate distance between the left optical center and the projection point, is the horizontal coordinate distance between the right optical center and the corresponding projection point, is the parallax distance, The distance between the center points is calculated based on the theory of similar triangles, the ratio of the depth value to the focal length, and the ratio of the optical center distance to the distance between the center points. The corresponding depth value is then used to complete the recognition and ranging of the detection target and obtain the target detection result.

[0160] In the specific implementation process, the overall process of this application may include: performing multi-target detection on the binocular image through the target detection algorithm YOLOX algorithm, extracting features of different scales in the binocular image respectively, and fusing the extracted multi-scale features to obtain features that are easier to detect, obtaining the length, width and coordinate position of the detection frame through the detection head, extracting a preset number of feature points in the detection frame and calculating the corresponding descriptors to complete the target matching of the dual-view image, based on the matched detection frame information, using the binocular imaging principle to calculate the target depth to complete the ranging function, and obtain the corresponding target detection result.

[0161] In the above embodiment, by obtaining the target detection frames of the two perspectives in the binocular image, and determining the calibration feature points at the predetermined positions corresponding to each target detection frame, the features of each detection frame are obtained based on the calibration feature points, and then the target detection results of the scene to be identified are obtained based on the features corresponding to the first perspective detection frame and the second perspective detection frame. The calibration feature points of the detection frames of different perspectives are used to identify the targets in the image, and accurate target detection results can be calculated in a timely manner with a small amount of data. This is suitable for portable devices with small computing power in special scenarios such as subways, and effectively improves the efficiency of target detection.

[0162] Furthermore, the pixel coordinate system corresponding to the first-perspective detection frame is determined based on the first-perspective image, and the pixel coordinate system corresponding to the second-perspective detection frame is determined based on the second-perspective image, and the coordinate information of multiple calibrated feature points corresponding to each detection frame and the corresponding feature descriptors are obtained. By comparing the feature descriptors corresponding to the multiple calibrated feature points of two different perspectives, more accurate target recognition results can be obtained.

[0163] In addition, the first feature and the second feature corresponding to the first perspective detection frame and the second perspective detection frame are determined according to the feature calibration points, and matching is performed based on the features to obtain a successfully matched associated feature pair, thereby determining the detection target and depth information corresponding to the detection frame, which can quickly and accurately realize the recognition and ranging of the detection target.

[0164] In one example, the binocular vision-based target detection method of the present application is as follows: Figure 8 As shown, this may include:

[0165] Collecting binocular images of the scene to be identified; the binocular images include first-view images and second-view images;

[0166] Performing preliminary target detection on the first-view image in the binocular image to obtain at least one first-view detection frame; performing preliminary target detection on the second-view image in the binocular image to obtain at least one second-view detection frame;

[0167] Acquire multiple calibration feature points in the first viewing angle detection frame and the second viewing angle detection frame respectively, wherein the relative positions of the calibration feature points in the first viewing angle detection frame and the second viewing angle detection frame are fixed;

[0168] Establishing a pixel coordinate system on the first-view image and the second-view image;

[0169] Determine the coordinates of a plurality of calibrated feature points of the first viewing angle detection frame in a pixel coordinate system to obtain a first feature; and obtain a first feature descriptor based on the coordinates of each calibrated feature point in the first feature;

[0170] Determine the coordinates of a plurality of calibrated feature points of the second viewing angle detection frame in a pixel coordinate system to obtain a second feature; and obtain a second feature descriptor based on the coordinates of each calibrated feature point in the second feature;

[0171] Matching the first feature descriptor with the second feature descriptor, and if the match is successful, obtaining an associated feature pair;

[0172] Determine a first center point of a first-view detection frame corresponding to the associated feature pair;

[0173] Determine a second center point of a second perspective detection frame corresponding to the associated feature pair;

[0174] Determining depth information of the detection target based on the first center point and the second center point;

[0175] Based on the depth information, the target detection result of the scene to be identified is obtained.

[0176] The above-mentioned target detection method based on binocular vision obtains the target detection frames of the two perspectives in the binocular image, determines the calibration feature points at the predetermined positions corresponding to each target detection frame, obtains the features of each detection frame based on the calibration feature points, and thus obtains the target detection results of the scene to be identified based on the features corresponding to the first perspective detection frame and the second perspective detection frame. The calibrated feature points of the detection frames of different perspectives are used to identify the targets in the image. Accurate target detection results can be calculated in a timely manner with a small amount of data. It is suitable for portable devices with small computing power in special scenarios such as subways, and effectively improves the efficiency of target detection.

[0177] Furthermore, the pixel coordinate system corresponding to the first-perspective detection frame is determined based on the first-perspective image, and the pixel coordinate system corresponding to the second-perspective detection frame is determined based on the second-perspective image, and the coordinate information of multiple calibrated feature points corresponding to each detection frame and the corresponding feature descriptors are obtained. By comparing the feature descriptors corresponding to the multiple calibrated feature points of two different perspectives, more accurate target recognition results can be obtained.

[0178] In addition, the first feature and the second feature corresponding to the first perspective detection frame and the second perspective detection frame are determined according to the feature calibration points, and matching is performed based on the features to obtain a successfully matched associated feature pair, thereby determining the detection target and depth information corresponding to the detection frame, which can quickly and accurately realize the recognition and ranging of the detection target.

[0179] The present application embodiment provides a target detection device based on binocular vision, such as Figure 9 As shown, the binocular vision-based target detection device 90 may include: an acquisition module 901, a detection frame determination module 902, a feature point determination module 903, a feature acquisition module 904, and a result acquisition module 905, wherein:

[0180] The acquisition module 901 is used to acquire binocular images of the scene to be recognized;

[0181] A detection frame determination module 902 is configured to obtain at least one first-view detection frame and at least one second-view detection frame based on the binocular image;

[0182] A feature point determination module 903 is configured to obtain a plurality of calibration feature points in the first viewing angle detection frame and the second viewing angle detection frame, respectively, wherein the relative positions of the calibration feature points in the first viewing angle detection frame and the second viewing angle detection frame are fixed;

[0183] A feature acquisition module 904 is configured to determine a first feature and a second feature based on the calibrated feature points, wherein the first feature is a feature corresponding to the first viewing angle detection frame, and the second feature is a feature corresponding to the second viewing angle detection frame;

[0184] The result acquisition module 905 is used to obtain the target detection result of the scene to be identified based on the first feature and the second feature.

[0185] As an optional embodiment, in the apparatus, the detection frame determination module 902 is specifically configured to:

[0186] Performing preliminary target detection on the first-perspective image in the binocular image to obtain at least one first-perspective detection frame;

[0187] Perform preliminary target detection on the second-view image in the binocular image to obtain at least one second-view detection frame.

[0188] As an optional embodiment, in the device, the feature acquisition module 904 is specifically configured to:

[0189] Establishing pixel coordinate systems on the first-view image and the second-view image respectively;

[0190] Determine the coordinates of multiple calibrated feature points of the first viewing angle detection frame in the pixel coordinate system to obtain a first feature;

[0191] The coordinates of a plurality of calibrated feature points of the second viewing angle detection frame in the pixel coordinate system are determined to obtain a second feature.

[0192] As an optional embodiment, in the device, the feature acquisition module 904 is specifically configured to:

[0193] Obtaining a first feature descriptor based on the coordinates of each calibrated feature point in the first feature;

[0194] Obtaining a second feature descriptor based on the coordinates of each calibrated feature point in the second feature;

[0195] The first feature descriptor and the second feature descriptor are compared to obtain an object detection result of the scene to be identified.

[0196] As an optional embodiment, in the device, the result acquisition module 905 is specifically configured to:

[0197] Matching the first feature with the second feature to obtain a matching result;

[0198] Based on the matching results, the target detection results of the scene to be identified are obtained.

[0199] As an optional embodiment, in the device, the result acquisition module 905 is specifically configured to:

[0200] For each first feature, matching each second feature with the first feature one by one;

[0201] If the match is successful, the first feature is associated with the second feature to obtain an associated feature pair.

[0202] As an optional embodiment, in the device, the result acquisition module 905 is specifically configured to:

[0203] Determine the first-view detection frame and the second-view detection frame corresponding to the associated feature pair respectively;

[0204] Determine the corresponding detection target based on the first perspective detection frame and the second perspective detection frame corresponding to the associated features;

[0205] Based on the detected target, the target detection result of the scene to be identified is obtained.

[0206] As an optional embodiment, in the device, the result acquisition module 905 is specifically configured to:

[0207] Determine a first center point of a first-view detection frame corresponding to the associated feature pair;

[0208] Determine a second center point of a second perspective detection frame corresponding to the associated feature pair;

[0209] Determining depth information of the detection target based on the first center point and the second center point;

[0210] Based on the depth information, the target detection result of the scene to be identified is obtained.

[0211] The binocular vision-based target detection device provided by the present application obtains target detection frames of the two perspectives in the binocular image, determines calibration feature points at predetermined positions corresponding to each target detection frame, obtains features of each detection frame based on the calibration feature points, and thus obtains target detection results of the scene to be identified based on features corresponding to the first perspective detection frame and the second perspective detection frame. The calibrated feature points of the detection frames of different perspectives are used to identify targets in the image, and accurate target detection results can be calculated in a timely manner with a small amount of data. The device is suitable for portable devices with small computing power in special scenarios such as subways, and effectively improves the efficiency of target detection.

[0212] Furthermore, the pixel coordinate system corresponding to the first-perspective detection frame is determined based on the first-perspective image, and the pixel coordinate system corresponding to the second-perspective detection frame is determined based on the second-perspective image, and the coordinate information of multiple calibrated feature points corresponding to each detection frame and the corresponding feature descriptors are obtained. By comparing the feature descriptors corresponding to the multiple calibrated feature points of two different perspectives, more accurate target recognition results can be obtained.

[0213] In addition, the first feature and the second feature corresponding to the first perspective detection frame and the second perspective detection frame are determined according to the feature calibration points, and matching is performed based on the features to obtain a successfully matched associated feature pair, thereby determining the detection target and depth information corresponding to the detection frame, which can quickly and accurately realize the recognition and ranging of the detection target.

[0214] The devices of the embodiments of the present application can execute the methods provided in the embodiments of the present application, and their implementation principles are similar and have corresponding technical effects. The actions performed by each module in the devices of the embodiments of the present application correspond to the steps in the methods of the embodiments of the present application. For detailed functional descriptions of each module of the device, please refer to the descriptions of the corresponding methods shown above, and will not be repeated here.

[0215] In an embodiment of the present application, an electronic device (computer device / equipment / system) is provided, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method provided in any optional embodiment of the present application. Compared with the prior art, the following can be achieved: by obtaining target detection frames of two perspectives in a binocular image, and determining calibration feature points at predetermined positions corresponding to each target detection frame, the features of each detection frame are obtained based on the calibration feature points, and the target detection results of the scene to be identified are obtained based on the features corresponding to the first perspective detection frame and the second perspective detection frame. The calibrated feature points of the detection frames of different perspectives are used to identify the target in the image, and accurate target detection results can be calculated in a timely manner with a small amount of data. This is suitable for portable devices with small computing power in special scenarios such as subways, and effectively improves the efficiency of target detection.

[0216] In an alternative embodiment, an electronic device is provided, such as Figure 10 As shown, Figure 10 The electronic device 1000 shown includes: a processor 1001 and a memory 1003. The processor 1001 and the memory 1003 are connected, for example, via a bus 1002. Optionally, the electronic device 1000 may further include a transceiver 1004, which may be used for data exchange between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the number of transceivers 1004 is not limited to one, and the structure of the electronic device 1000 does not constitute a limitation on the embodiments of the present application.

[0217] Processor 1001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic device, transistor logic device, hardware component, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 1001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, or a combination of a DSP and a microprocessor.

[0218] Bus 1002 may include a path for transmitting information between the above components. Bus 1002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. Bus 1002 may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, Figure 10 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0219] The memory 1003 may be a ROM (Read Only Memory) or other type of static storage device that can store static information and instructions, a RAM (Random Access Memory) or other type of dynamic storage device that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium that can be used to carry or store computer programs and can be read by a computer, without limitation herein.

[0220] The memory 1003 is used to store the computer program for executing the embodiments of the present application, and the execution is controlled by the processor 1001. The processor 1001 is used to execute the computer program stored in the memory 1003 to implement the steps shown in the above method embodiments.

[0221] Among them, electronic devices include but are not limited to: portable devices or other electronic devices that can be used in subways, transportation, industrial manufacturing or other scenarios with low computing power.

[0222] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps and corresponding contents of the aforementioned method embodiment can be implemented.

[0223] It should be noted that the computer-readable storage medium mentioned above in this application may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0224] An embodiment of the present application also provides a computer program product, including a computer program, which can implement the steps and corresponding contents of the aforementioned method embodiment when executed by a processor.

[0225] In the specification and claims of this application and the accompanying drawings, the terms "first," "second," "third," "fourth," "1," "2," and so on (if any) are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that the terms used in this manner are interchangeable where appropriate, such that the embodiments of the present application described herein can be practiced in an order other than that shown or described.

[0226] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0227] It should be understood that, although each operation step is indicated by arrows in the flowchart of the embodiment of the present application, the order of implementation of these steps is not limited to the order indicated by the arrows. Unless otherwise clearly stated herein, in some implementation scenarios of the embodiment of the present application, the implementation steps in each flowchart can be performed in other orders according to demand. In addition, some or all of the steps in each flowchart can include multiple sub-steps or multiple stages based on actual implementation scenarios. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage in these sub-steps or stages can also be executed at different times respectively. Under different scenarios at the execution time, the execution order of these sub-steps or stages can be flexibly configured according to demand, and the embodiment of the present application does not limit this.

[0228] The above description is only an optional implementation method for some implementation scenarios of this application. It should be pointed out that for ordinary technicians in this technical field, without departing from the technical concept of the solution of this application, the use of other similar implementation methods based on the technical ideas of this application also falls within the protection scope of the embodiments of this application.

Claims

1. A target detection method based on binocular vision, characterized in that: include: Collect binocular images of the scene to be identified; The scene to be identified includes a subway scene; Based on the binocular image, obtaining at least one first-viewing angle detection frame and at least one second-viewing angle detection frame, wherein the binocular image includes the first-viewing angle image and the second-viewing angle image; Acquire multiple calibration feature points in the first and second perspective detection frames, respectively, where each calibration feature point is fixed relative to the first and second perspective detection frames; the calibration feature points include points on the detection frame lines and points within the detection frames; the points within the detection frames include the center point of the detection frame and points that are shifted a certain distance outward from the center point of the detection frame; Determining a first feature and a second feature according to the calibrated feature points, respectively, where the first feature is a feature corresponding to the first viewing angle detection frame, and the second feature is a feature corresponding to the second viewing angle detection frame; The determining of the first feature and the second feature respectively according to the calibrated feature points includes: Based on the calibrated feature points corresponding to the first viewing angle detection frame, determine the pixels within a certain range around each calibrated feature point, analyze the features of the pixels within the range, and determine the first feature; based on the calibrated feature points corresponding to the second viewing angle detection frame, determine the pixels within a certain range around each calibrated feature point, analyze the features of the pixels within the range, and determine the second feature; Based on the first feature and the second feature, an object detection result of the scene to be identified is obtained.

2. The binocular vision-based target detection method according to claim 1, characterized in that: The obtaining, based on the binocular image, at least one first-viewing angle detection frame and at least one second-viewing angle detection frame comprises: Performing preliminary target detection on the first-view image in the binocular image to obtain the at least one first-view detection frame; Perform preliminary target detection on the second perspective image in the binocular image to obtain the at least one second perspective detection frame.

3. The binocular vision-based target detection method according to claim 2, characterized in that: The determining of the first feature and the second feature respectively according to the calibrated feature points includes: Establishing pixel coordinate systems on the first perspective image and the second perspective image respectively; determining coordinates of a plurality of calibrated feature points of the first viewing angle detection frame in the pixel coordinate system to obtain the first feature; Determine the coordinates of a plurality of calibrated feature points of the second viewing angle detection frame in the pixel coordinate system to obtain the second feature.

4. The binocular vision-based target detection method according to claim 3, characterized in that: The obtaining, based on the first feature and the second feature, a target detection result of the scene to be identified, includes: Obtaining a first feature descriptor based on the coordinates of each calibrated feature point in the first feature; Obtaining a second feature descriptor based on the coordinates of each calibrated feature point in the second feature; The first feature descriptor and the second feature descriptor are compared to obtain an object detection result of the scene to be identified.

5. The binocular vision-based target detection method according to claim 1, wherein: The obtaining, based on the first feature and the second feature, a target detection result of the scene to be identified, includes: Matching the first feature with the second feature to obtain a matching result; Based on the matching result, an object detection result of the scene to be identified is obtained.

6. The binocular vision-based target detection method according to claim 5, characterized in that: The matching the first feature with the second feature to obtain a matching result includes: For each first feature, matching each of the second features with the first feature one by one; If the match is successful, the first feature is associated with the second feature to obtain an associated feature pair.

7. The binocular vision-based target detection method according to claim 6, characterized in that: Obtaining a target detection result of the scene to be identified based on the matching result includes: respectively determining a first perspective detection frame and a second perspective detection frame corresponding to the associated feature pair; Determine a corresponding detection target based on the first perspective detection frame and the second perspective detection frame corresponding to the associated feature pair; Based on the detected target, a target detection result of the scene to be identified is obtained.

8. The binocular vision-based target detection method according to claim 7, characterized in that: Obtaining a target detection result of the scene to be identified based on the detected target includes: Determine a first center point of a first viewing angle detection frame corresponding to the associated feature pair; Determining a second center point of a second viewing angle detection frame corresponding to the associated feature pair; Determining depth information of the detection target based on the first center point and the second center point; Based on the depth information, an object detection result of the scene to be identified is obtained.

9. A target detection device based on binocular vision, characterized in that: include: An acquisition module, used to acquire binocular images of the scene to be identified; The scene to be identified includes a subway scene; a detection frame determination module, configured to obtain at least one first-viewing angle detection frame and at least one second-viewing angle detection frame based on the binocular image, wherein the binocular image includes the first-viewing angle image and the second-viewing angle image; a feature point determination module, configured to obtain a plurality of calibration feature points in each of the first and second perspective detection frames, wherein each calibration feature point has a fixed relative position in the first and second perspective detection frames; the calibration feature points include points on the detection frame line and points within the detection frame; the points within the detection frame include the center point of the detection frame and points shifted a certain distance outward from the center point of the detection frame; A feature acquisition module is configured to determine a first feature and a second feature based on the calibrated feature points, wherein the first feature is a feature corresponding to the first viewing angle detection frame, and the second feature is a feature corresponding to the second viewing angle detection frame. The determining of the first feature and the second feature based on the calibrated feature points includes: determining pixels within a certain range around each calibrated feature point based on the calibrated feature points corresponding to the first viewing angle detection frame, analyzing features of the pixels within the range, and determining the first feature; and determining pixels within a certain range around each calibrated feature point based on the calibrated feature points corresponding to the second viewing angle detection frame, analyzing features of the pixels within the range, and determining the second feature. A result acquisition module is used to obtain the target detection result of the scene to be identified based on the first feature and the second feature.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory, wherein: The processor executes the computer program to implement the steps of the binocular vision-based target detection method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the binocular vision-based target detection method according to any one of claims 1 to 8 are implemented.

12. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the binocular vision-based target detection method according to any one of claims 1 to 8 are implemented.

Images