Distance measurement positioning method and system, storage medium, electronic equipment and vehicle

Abstract

The invention relates to a distance measurement positioning method and system, a storage medium, electronic equipment and a vehicle, and the method comprises the steps: carrying out the processing of a collected initial monocular image through a trained target detection model, and obtaining a detection object; if the target object exists in the detection objects, obtaining initial depth information of the target object; intercepting a target area from the binocular image of the target object based on the initial depth information; and determining target depth information of the target object based on the target area. According to the method, through the improved target detection model, fine positioning of the object is realized through coarse positioning, the positioning efficiency and precision are improved, and the method is suitable for being deployed on edge equipment.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and in particular to a ranging and positioning method, system, storage medium, electronic device and vehicle. Background Art

[0002] When currently AGVs (Automated Guided Vehicles) are used in factories and warehouses, although they have certain autonomous navigation capabilities, during the operation of the AGV, the internal data processing logic may be defective or insufficient, or changes in the external environment, such as the sudden appearance of obstacles, changes in light conditions, wear and tear of ground markings, or other unforeseen factors, may cause the AGV to lose its path tracking ability and be unable to continue to perform tasks according to the scheduled route, requiring manual intervention to recalibrate or troubleshoot. Therefore, it is necessary to accurately locate the AGV to solve the navigation loss problem that may be encountered during its operation. Summary of the invention

[0003] The embodiments of the present application provide a ranging and positioning method, system, storage medium, electronic device and vehicle to solve the problem of AGV positioning.

[0004] In order to achieve the above object, according to the first aspect of the present application, a ranging-based positioning system is provided, and the method includes:

[0005] The collected initial monocular image is processed by the trained target detection model to obtain the detected object;

[0006] If there is a target object in the detected object, obtaining initial depth information of the target object;

[0007] Based on the initial depth information, intercepting a target area from the binocular image of the target object;

[0008] Based on the target area, target depth information of the target object is determined.

[0009] Optionally, the acquiring initial depth information of the target object includes:

[0010] Respectively acquiring a first monocular image and a second monocular image of the target object at different times;

[0011] Initial depth information of the target object is acquired based on the first monocular image and the second monocular image.

[0012] Optionally, acquiring initial depth information of the target object based on the first monocular image and the second monocular image includes:

[0013] Inputting the first monocular image and the second monocular image into the target detection model respectively, obtaining the pixel coordinates of the target object in the first monocular image and the pixel coordinates of the target object in the second monocular image;

[0014] Determine a motion vector of the target object based on pixel coordinates in the first monocular image and pixel coordinates in the second monocular image;

[0015] Initial depth information of the target object is acquired based on the motion vector and pixel coordinates in the first monocular image.

[0016] Optionally, acquiring initial depth information of the target object based on the motion vector and pixel coordinates in the first monocular image includes:

[0017] Predicting, based on the motion vector and the pixel coordinates in the first monocular image, a first pixel coordinate of the target object in a third monocular image, where the third monocular image is an image taken at the same time point as the first monocular image but from a different perspective;

[0018] Initial depth information of the target object is acquired based on first pixel coordinates in the third monocular image and pixel coordinates in the first monocular image.

[0019] Optionally, acquiring initial depth information of the target object based on first pixel coordinates in the third monocular image and pixel coordinates in the first monocular image includes:

[0020] Optimizing the first pixel coordinates in the third monocular image based on the actual pixel coordinates in the third monocular image to determine the second pixel coordinates of the target object in the third monocular image;

[0021] Initial depth information of the target object is calculated based on the second pixel coordinates in the third monocular image and the pixel coordinates in the first monocular image, and camera calibration parameters.

[0022] Optionally, determining target depth information of the target object based on the target area includes:

[0023] Inputting the target area into a target detection model to obtain pixel coordinates of the target object in the target area;

[0024] Based on the pixel coordinates of the target object in the target area, target depth information of the target object is determined.

[0025] Optionally, determining target depth information of the target object based on pixel coordinates of the target object in the target area includes:

[0026] Based on the pixel coordinates of the target object in the target area and the camera calibration parameters, target depth information of the target object is determined.

[0027] Optionally, determining the target depth information of the target object based on the pixel coordinates of the target object in the target area and a camera calibration parameter includes:

[0028] Determining the disparity of the target object in the target area based on the pixel coordinates of the target area;

[0029] The product of the focal length and the center distance of the camera calibration parameters is calculated, and the target depth information of the target object is determined based on the ratio of the product to the parallax.

[0030] Optionally, after determining the target depth information of the target object based on the target area, the method further includes:

[0031] If the target object includes a target vehicle and a target cargo, driving planning is performed on the target vehicle based on the target depth information of the target vehicle and the target depth information of the target cargo.

[0032] Optionally, the method further comprises:

[0033] Acquire multiple chessboard images of preset chessboard grids at different angles and orientations;

[0034] Determine the pixel coordinates of each corner point in the chessboard image;

[0035] Determine the projection coordinates of the corner point in the world coordinate system;

[0036] A camera calibration parameter is determined based on the projection coordinates and the pixel coordinates.

[0037] Optionally, the target detection model includes a first network, a second network and a third network connected in sequence;

[0038] The method processes the collected initial monocular image by using the trained target detection model to obtain the detected object, including:

[0039] Using the optimization module of the first network to extract features of the target object in the monocular image, and obtain feature information of multiple scales;

[0040] Determine the weight of each of the feature information by using the attention mechanism of the second network, and enhance the features of the plurality of feature information based on the weights to obtain a plurality of enhanced features; and obtain a plurality of fused features based on the plurality of enhanced features;

[0041] The third network is used to process the multiple fusion features, determine whether the detection object exists in the initial monocular image, and determine whether the detection object exists in the monocular image, and mark the detection frame of the detection object in the monocular image.

[0042] Optionally, the using the optimization module of the first network to extract features of the target object in the monocular image to obtain feature information of multiple scales includes:

[0043] The input initial monocular image is downsampled in sequence through a plurality of sequentially connected optimization modules to obtain the feature information of the plurality of different scales.

[0044] Optionally, the attention mechanism is a grouped attention module,

[0045] The using the attention mechanism of the second network to determine the weight of each of the feature information, and enhancing the multiple feature information based on the weights to obtain multiple enhanced features, including:

[0046] Based on the grouped attention module, the weights of the input feature information in multiple channels are determined, and the weights of the multiple channels are weighted to determine multiple channel attention features of the feature information; feature fusion and residual connection are performed on the multiple channel attention features to obtain the enhanced features.

[0047] Optionally, determining the weights of the input feature information in multiple channels based on the grouped attention module includes:

[0048] Perform a segmentation operation on the feature map of the input group attention module to obtain a feature map of the first branch and a feature map of the second branch;

[0049] Performing different processing on the feature graph of the first branch and the feature graph of the second branch respectively to obtain a processed feature graph of the first branch and a processed feature graph of the second branch;

[0050] Based on the processed feature map of the first branch and the processed feature map of the second branch, weights are extracted through a channel attention module, and the channel weights of the first branch and the channel weights of the second branch are output.

[0051] Optionally, extracting weights through a channel attention module based on the processed feature map of the first branch and the processed feature map of the second branch, and outputting the channel weight of the first branch and the channel weight of the second branch include:

[0052] Superimposing the processed feature map of the first branch and the processed feature map of the second branch to obtain a fused feature map;

[0053] Performing a pooling operation on the fused feature map to obtain global information of multiple channels;

[0054] Based on the global information of the multiple channels, a channel weight of the first branch and a channel weight of the second branch are determined.

[0055] Optionally, performing a pooling operation on the fused feature map to obtain global information of multiple channels includes:

[0056] The fused feature map is input into the pooling formula to obtain the global information of the multiple channels. The pooling formula is:

[0057]

[0058] Among them, S c is the global information of the cth channel, is the fused feature map, H and W are the height and width of the fused feature map respectively, and C is the number of channels.

[0059] Optionally, determining the channel weight of the first branch and the channel weight of the second branch based on the global information of the multiple channels includes:

[0060] The global information of the multiple channels is sequentially subjected to convolution operations of two convolutional layers to obtain local information of the first branch and local information of the second branch;

[0061] The local information of the first branch and the local information of the second branch are normalized to obtain the channel weight of the first branch and the channel weight of the second branch. The formula for the normalization is:

[0062]

[0063] in, is the local information of the i-th branch in the c-th channel, i = 1, 2; a i (c) is the channel weight of the i-th branch in the c-th channel.

[0064] According to a second aspect of the present application, a ranging and positioning system is provided, the system comprising:

[0065] The detection module is used to process the collected initial monocular image through the trained target detection model to obtain the detected object;

[0066] An acquisition module, configured to acquire initial depth information of a target object if there is a target object in the detected object;

[0067] A capture module, configured to capture a target area from the binocular image of the target object based on the initial depth information;

[0068] A positioning module is used to determine target depth information of the target object based on the target area.

[0069] According to a third aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored. The computer-readable storage medium stores instructions, and when the instructions are executed by a computer, the computer implements any one of the ranging and positioning methods provided in the embodiments of the present application.

[0070] According to a fourth aspect of the present application, there is provided an electronic device, including:

[0071] a memory having a computer program stored thereon;

[0072] A processor is used to execute the computer program in the memory to implement any ranging and positioning method provided in the embodiments of the present application.

[0073] According to a fifth aspect of the present application, a vehicle is provided, comprising the ranging and positioning system.

[0074] Some embodiments of the present specification include at least the following beneficial effects: on the basis of coarse positioning, more accurate positioning information is obtained through target detection models and binocular ranging principles, which is conducive to improving the positioning effect, and this method has low cost, fast processing speed, and lightweight model, and is more suitable for deployment on edge devices.

[0075] Other features and advantages of the present application will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application, and those skilled in the art can obtain other drawings based on these drawings without creative work.

[0077] In order to more completely understand the present application and its beneficial effects, the following description will be given in conjunction with the accompanying drawings, wherein the same figure numbers represent the same parts in the following description.

[0078] Figure 1 is an application scenario diagram of a ranging and positioning system according to some embodiments of this specification;

[0079] Figure 2 is an exemplary flow chart of a ranging and positioning method according to some embodiments of this specification;

[0080] Figure 3 is an exemplary flow chart of determining a detection object according to some embodiments of this specification;

[0081] Figure 4 is an exemplary schematic diagram of a target detection model according to some embodiments of this specification;

[0082] Figure 5 is an exemplary schematic diagram of an optimization module according to some embodiments of this specification;

[0083] Figure 6 is an exemplary schematic diagram of a grouping attention module according to some embodiments of the present specification;

[0084] Figure 7 is an exemplary schematic diagram of another grouping attention module according to some embodiments of the present specification;

[0085] Figure 8 is an exemplary structural diagram of a ranging and positioning system according to some embodiments of this specification;

[0086] Fig. 9 It is a schematic diagram of the structure of an electronic device according to some embodiments of this specification. DETAILED DESCRIPTION

[0087] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0088] In order to facilitate understanding of the implementation scheme provided in the embodiment of the present application, the relevant application background of the ranging and positioning system provided in the embodiment of the present application is first described.

[0089] At present, in order to solve the positioning problem of AGV, it is proposed to use binocular cameras, traditional image processing algorithms (such as Canny edge detection, template matching, etc.) or target detection algorithms based on deep learning models (such as Nanodet) for target detection and positioning. However, the use of traditional image processing algorithms is easily interfered by the environment and similar objects, and template matching is time-consuming, which affects the accuracy of ranging and positioning. Moreover, the Nanodet model has low detection accuracy for small targets and poor ability to distinguish small targets and smaller targets. In an environment with poor image quality, the specific location of the target object cannot be identified, and there are still problems of missed detection and false detection in the field of small target detection. The deep learning model deployed on the camera may need to process various video data. As an edge computing device, the camera has weak computing power, which may cause blurred monitoring images in special scenarios, leading to problems of misrecognition.

[0090] In view of this, some embodiments of this specification provide a ranging positioning system, which is deployed based on an improved Nanodet model to process the video stream of a binocular camera, realize real-time target tracking and distance measurement of AGV, and improve positioning accuracy.

[0091] The system and method of the present application can be applied to various scenarios of automated environments that require precise positioning and navigation, such as industrial manufacturing, logistics warehousing, and healthcare. By detecting and locating specific raw material stacking points or finished product stacking points, AGV can accurately complete the task of transporting materials.

[0092] It should be noted that the description order of the following embodiments is not intended to limit the preferred order of the embodiments. Although the logical order is shown in the flow chart, in some cases, the steps shown or described may be performed in an order different from that shown in the drawings.

[0093] Figure 1 This is an application scenario diagram of the ranging and positioning method shown in some embodiments of this specification.

[0094] like Figure 1 As shown, the application scenario 100 based on the ranging and positioning system may include a processor 110 , a network 120 , a vehicle 130 , and an image acquisition device 140 .

[0095] like Figure 1 As shown, the application scenario 100 of the ranging and positioning system may include a processor 110 , a network 120 , a vehicle 130 , and an image acquisition device 140 .

[0096] The processor 110 may be a local physical server, a cloud server (such as Figure 1 as shown), a virtual server, a distributed server, or any other suitable computing device.

[0097] In some embodiments, the processor 110 may be integrated into the image acquisition device 140 .

[0098] In some embodiments, the processor 110 may acquire data of the surrounding environment. The processor 110 may use the acquired data to determine the position information of various objects in the surrounding environment. For example, the processor 110 may obtain data from the image acquisition device 140. The processor 110 may communicate with the image acquisition device 140 and / or other components of the vehicle 130 via the network 120.

[0099] The network 120 may include but is not limited to any one or combination of wireless local area network (WLAN), wide area network (WAN), wireless network of radio wave / cellular network / satellite communication network and / or local wireless network or short-range wireless network (Bluetooth™).

[0100] In some embodiments, the processor 110 may process the collected initial monocular image based on the trained target detection model to obtain the detected object; if there is a target object in the detected object, the initial depth information of the target object is obtained; based on the initial depth information, the target area is intercepted from the binocular image of the target object; based on the target area, the target depth information of the target object is determined. For more information about this embodiment, please refer to Figure 2 Related description.

[0101] The vehicle 130 may be a transportation service vehicle for the application scenario 100. In some embodiments, the vehicle 130 may include a taxi, a private car, a ride-sharing car, a shared vehicle, a bus bicycle, a tricycle, a motorcycle, a robot, an AGV, etc., or any combination thereof. In some embodiments, the vehicle 130 may include a private self-driving car, a shared self-driving car, an unmanned automatic freight car, etc. In some embodiments, the vehicle 130 may receive instructions issued by the processor 110 and complete corresponding tasks in accordance with the instructions. For example, the vehicle 130 may receive instructions such as performing an autonomous driving operation and automatically control the vehicle 130 to complete the corresponding operation.

[0102] The image acquisition device 140 is used to capture data of the surrounding environment. In some embodiments, the image acquisition device 140 includes but is not limited to cameras, webcams, video cameras, other devices with camera functions (mobile phones, tablets, etc.), etc. The camera may include a monocular camera, a binocular camera, a depth camera, etc. The camera may also be two cameras (such as a first monocular camera, a second monocular camera, etc.) with an optical system installed side by side and aligned to point in the same direction. Since the binocular camera has two viewpoints, the image taken by the binocular camera contains depth information. It is conceivable that other suitable cameras or sensors capable of sensing depth information may also be used. The image acquisition device 140 can record a video consisting of multiple image frames captured at multiple time points. In some embodiments, the image acquisition device 140 can capture a series of binocular image frames of a scene. The binocular image frames can be sent to the processor 110 in real time (e.g., by streaming), or sent to the processor 110 together after the vehicle 130 completes a portion of the trajectory.

[0103] In some embodiments, the image acquisition device 140 can be a monitoring device in the environment, and the image acquisition device 140 can also be installed on the target environment based on a mounting structure. In some embodiments, the mounting structure can use screws, adhesives or other mounting structures. In some embodiments, any suitable mounting structure can also be used.

[0104] In some embodiments, the image acquisition device 140 may be deployed at least one location in the target environment. In some embodiments, the number and location of the image acquisition device 140 may be preset. For example, the target environment may be divided to obtain multiple areas of the target environment, and each area may be correspondingly installed with the image acquisition device 140. The target environment may be the environment where the image acquisition device is located, such as a factory, a warehouse, or a logistics park.

[0105] In some embodiments, the image acquisition device 140 may communicate with one or more components in the application scenario (eg, the vehicle 130 , the processor 110 ) via the network 120 .

[0106] It should be noted that application scenario 100 is provided only for the purpose of illustration and is not intended to limit the scope of this specification. For those of ordinary skill in the art, various modifications or changes can be made according to the description of this specification. For example, application scenario 100 can also include a database, an information source, etc. For another example, application scenario 100 can be implemented on other devices to achieve similar or different functions. However, changes and modifications will not deviate from the scope of this specification.

[0107] Figure 2 It is an exemplary flow chart of the ranging and positioning method shown in some embodiments of this specification.

[0108] In some embodiments, process 200 may be implemented by a processor based on a ranging positioning system. Figure 2 As shown, process 200 includes the following steps:

[0109] Step S210, processing the collected initial monocular image by using the trained target detection model to obtain the detected object.

[0110] The initial monocular image refers to an image of the surrounding environment captured by an image acquisition device in the environment.

[0111] In some embodiments, the initial monocular image may be an image captured by a monocular camera (e.g., a first monocular camera). In some embodiments, the initial monocular image may also be an image captured by a certain camera in a binocular camera. For example, the monocular image may be a left-eye image or a right-eye image.

[0112] For the purpose of easy distinction, the two cameras in the binocular camera are taken as a left camera and a right camera as examples.

[0113] The left-eye image is the image captured by the left-eye camera under the preset field of view.

[0114] The right-eye image is the image captured by the right-eye camera under the preset field of view.

[0115] The preset field of view is the range of view that the camera can capture under specific settings.

[0116] In some embodiments, the left-eye image and the right-eye image are images taken by the left-eye camera and the right-eye camera respectively at the same time or a similar time.

[0117] The detected object refers to the specific object identified in the initial monocular image. For example, the detected object may include other vehicles, goods, obstacles, people, etc.

[0118] In some embodiments, the processor can be connected in communication with the image acquisition device to obtain an initial monocular image. For example, during idle time (e.g., the time period when the AGV waits for the next task after completing a handling task), the processor can control the binocular camera to turn on only one of the cameras (e.g., the left camera), and the processor can acquire the image sent by the camera over time as the initial monocular image to detect whether there is a target object in the environment and track it. For example, the left camera can communicate with the processor through a network to send the initial monocular image continuously, periodically, or intermittently. In some embodiments, the processor can also be connected in communication with other image acquisition devices (e.g., the first monocular camera, etc.) to obtain an initial monocular image.

[0119] The target prediction model is a model for obtaining the location and / or category information of an object. In some embodiments, the target prediction model may be a machine learning model, which may include but is not limited to any one or combination of a convolutional neural network (CNN) model, a neural network (NN) model, or other custom model structures.

[0120] In some embodiments, the input of the target prediction model includes an image to be detected, and the output is a detection result, which may include the detection box coordinates and / or category information of the detected object in the image to be detected.

[0121] The image to be detected may be an image captured by an image acquisition device. For example, the image to be detected may include a monocular image, a left-eye image in a binocular image, a right-eye image in a binocular image, etc. For more information about binocular images, see Figure 2 The relevant description is as follows.

[0122] The image to be detected reflects information related to the environment in which the binocular camera is located. For example, the image to be detected may include information about objects around the binocular camera, such as dynamic object information (e.g., other vehicles, pedestrians, animals, etc.) and static object information (e.g., buildings, roads, traffic signs, goods, etc.). In some embodiments, the static objects may include planar objects (e.g., lane boundaries, lane centerlines, etc.) or non-planar objects (e.g., buildings, goods, etc.).

[0123] The image to be detected may be an image directly acquired by an image acquisition device, or may be an image acquired by a method such as video frame extraction from a video acquired by the image acquisition device.

[0124] In some embodiments, the target prediction model can be trained by a large number of training data sets with training labels. In some embodiments, the training can be performed based on the training data sets by various methods. For example, the training can be performed based on the gradient descent method.

[0125] In some embodiments, the target prediction model can also be obtained based on other network forms and corresponding network training. For example, the training of the target prediction model can be implemented by a Nanodet model, and the Nanodet model can be configured to be obtained by training based on a training data set. For specific model content, see Figure 3 Description of the relevant part.

[0126] Step S220: If there is a target object among the detected objects, initial depth information of the target object is obtained.

[0127] The target object refers to a specific object that needs to be identified and located in the image to be detected. For example, the target object may include an AGV vehicle or other vehicles that require auxiliary positioning, and the target object may also include cargo, other obstacles, etc.

[0128] In some embodiments, the processor may filter out detection objects of a target category from the detection results as target objects.

[0129] The detection objects of the target category refer to objects that can help determine the position of the vehicle or have a direct impact on the operation of the vehicle. For example, the detection objects of the target category may include target vehicles, obstacles, cargo, etc.

[0130] The target vehicle refers to a vehicle that needs to be located. For example, a vehicle may send a location request to the processor, and the processor may determine the vehicle as a target vehicle based on the location request.

[0131] The initial depth information refers to the predetermined or estimated depth information between the target object and the image acquisition device (eg, a binocular camera, etc.).

[0132] Depth information refers to the distance information of an object in three-dimensional space. For example, the depth information may include the actual distance and direction from a camera to a certain point on the surface of the object.

[0133] In some embodiments, the processor can determine the initial depth information of the target object in a variety of ways. For example, the processor can use a depth estimation model to predict a depth map from a single image and determine the initial depth information of the target object based on the depth map. The depth estimation model can be a neural network model, etc. The depth estimation model can be trained on a large amount of data to determine the approximate depth value of the target object. For another example, the processor can estimate the approximate depth value of the target object based on a single image, through geometric relationships and known camera calibration parameters, as the initial depth information of the target object.

[0134] In some embodiments, the processor may respectively obtain a first monocular image and a second monocular image of the target object at different times; and obtain initial depth information of the target object based on the first monocular image and the second monocular image.

[0135] The first monocular image refers to the image containing the target object captured by the image acquisition device at the first time.

[0136] The second monocular image refers to an image containing the target object acquired by the image acquisition device at a second time.

[0137] The first time may be a time point after the target object is recognized to exist in the detection object.

[0138] The second time may be any time point after the first time. For example, the second time may be the next moment of the first time, or may be a moment after a preset time interval with the first time.

[0139] In some embodiments, each frame of monocular image may be a real-time image. For example, the processor may acquire an image captured by an image acquisition device (e.g., a monocular camera or a binocular camera, etc.) in real time (e.g., every 0.02 seconds, 0.01 seconds, etc.). In some embodiments, each frame of monocular image may also be an image stored in a storage device.

[0140] In some embodiments, the processor may periodically acquire a monocular image video captured by an image acquisition device (e.g., a monocular camera or one of the binocular cameras, etc.), wherein the monocular image video is a sequence of multiple frames of monocular images in a time sequence of scanning. In some embodiments, each frame of the monocular image in the monocular image video may be time continuous.

[0141] In some embodiments, the processor may execute the ranging and positioning method shown in some embodiments of this specification on each frame of the monocular image in the monocular image video in chronological order.

[0142] In some embodiments, the processor can determine the initial depth information between the target object and the binocular camera based on a pre-calibrated correspondence of an image acquisition device (such as a binocular camera, etc.) and the pixel coordinates of the target object in the first monocular image and the second monocular image, wherein the pre-calibrated correspondence of the binocular camera is: the correspondence between the position points represented by each pixel coordinate in the image captured by the binocular camera and the actual distance between the binocular cameras.

[0143] In some embodiments, images of the surrounding environment can be captured by a binocular camera installed in a preset position. In order to ensure that the binocular camera can capture stable images, the binocular camera can be fixed by a gimbal. In addition, before the binocular camera captures images, the field of view direction of the camera needs to be adjusted and fixed to facilitate subsequent calibration of the captured images.

[0144] In some embodiments, target detection is performed on the first monocular image and the second monocular image captured by the binocular camera respectively, so as to obtain pixel coordinates of the target object in the first monocular image and pixel coordinates of the target object in the second monocular image.

[0145] In some embodiments, target detection on the first monocular image and the second monocular image can use a deep learning target detection and recognition algorithm to detect the pixel coordinates of the target object in the initial monocular image. In actual applications, the target object can be a vehicle that needs auxiliary positioning, or an obstacle on the current driving path, etc.

[0146] In some embodiments, the processor may determine a calibration image of the correspondence between the position points represented by each pre-calibrated pixel coordinate and the actual distance between the binocular camera, and then compare the first monocular image captured by the binocular camera with the calibration image, and determine the first depth information of the target object according to the pre-calibrated correspondence; compare the second monocular image captured by the binocular camera with the calibration image, and determine the second depth information of the target object according to the pre-calibrated correspondence; average the first depth information and the second depth information (e.g., arithmetic mean, weighted average, etc.) to determine the initial depth information of the target object.

[0147] In some embodiments of the present specification, the initial depth of the target object is determined by the first monocular image and the second monocular image. When the camera is fixed, the depth of the target object is inferred by the relative displacement of the target object in the two images, which can improve the performance and accuracy of subsequent binocular camera-based positioning.

[0148] In some embodiments, the processor may input the first monocular image and the second monocular image into the target detection model respectively to obtain the pixel coordinates of the target object in the first monocular image and the pixel coordinates of the target object in the second monocular image; determine the motion vector of the target object based on the pixel coordinates in the first monocular image and the pixel coordinates in the second monocular image; and obtain the initial depth information of the target object based on the motion vector and the pixel coordinates in the first monocular image.

[0149] Pixel coordinates are the coordinates of an object in the pixel coordinate system corresponding to the image.

[0150] In some embodiments, the target detection model may output the pixel coordinates of the four vertices of the detection box (e.g., the detection box of the target object) as pixel coordinates. In some embodiments, the pixel coordinates may also be the coordinates of the center point of the detection box, or the coordinates of the upper left corner, lower right corner, etc. of the detection box.

[0151] A motion vector refers to the temporal displacement vector of a pixel or feature point in an image.

[0152] In some embodiments, the processor may determine the motion vector of the target object in a variety of ways based on the pixel coordinates in the first monocular image and the pixel coordinates in the second monocular image. For example, the target detection model may be input based on the first monocular image and the second monocular image, respectively, to determine the pixel coordinates of the target object in the first monocular image and the pixel coordinates of the target object in the second monocular image; based on the pixel coordinates of the target object in the first monocular image and the pixel coordinates of the target object in the second monocular image, the difference between the pixel coordinates of the target pixel points in the two images is calculated to obtain the motion vector of the target object.

[0153] The motion vector reflects the difference between monocular images (e.g., left eye images) of the same object at different time points and different positions. For example, the motion vector may include the displacement of a pixel or feature point in the horizontal direction, or the displacement of a pixel or feature point in the vertical direction.

[0154] The target pixel point may be any point on the target object, for example, the target pixel point is the center point of the target object.

[0155] In some embodiments, the processor may calculate a first disparity based on the motion vector, and calculate depth information of the target object based on the first disparity and camera calibration parameters (eg, focal length and baseline distance, etc.).

[0156] The first disparity reflects the disparity of the target object in the first monocular image and the second monocular image.

[0157] In some embodiments, the processor may use the horizontal displacement of the pixel point or feature point of the motion vector as the first disparity.

[0158] In some embodiments of the present specification, initial depth information of the target object is estimated by using pixel coordinates in the first monocular image and pixel coordinates in the second monocular image, so as to improve the accuracy and robustness of subsequent positioning.

[0159] In some embodiments, the processor can predict the first pixel coordinates of the target object in a third monocular image based on the motion vector and the pixel coordinates in the first monocular image, where the third monocular image is an image of a different perspective taken at the same time point as the first monocular image; and obtain initial depth information of the target object based on the first pixel coordinates in the third monocular image and the pixel coordinates in the first monocular image.

[0160] The third monocular image refers to the image of the target object captured at the first moment at different angles and orientations.

[0161] In some embodiments, the third monocular image may form a corresponding binocular image with the first monocular image.

[0162] The first pixel coordinate refers to the estimated pixel coordinate of the target object in the third monocular image.

[0163] In some embodiments, the processor may predict the first pixel coordinates of the target object in the third monocular image in a variety of ways based on the motion vector and the pixel coordinates in the first monocular image. For example, the processor may determine the horizontal pixel coordinates of the target object in the third monocular image based on the sum of the horizontal displacement of the target pixel and the horizontal pixel coordinates of the target pixel in the first monocular image; and determine the vertical pixel coordinates of the target object in the third monocular image based on the sum of the vertical displacement of the target pixel and the vertical coordinates of the target pixel in the first monocular image.

[0164] In some embodiments, the processor may obtain initial depth information of the target object in a variety of ways based on the first pixel coordinates in the third monocular image and the pixel coordinates in the first monocular image. For example, the first pixel coordinates in the third monocular image are optimized based on a preset optimization value, the optimized first pixel coordinates are determined, and the initial depth information of the target object is obtained based on the optimized first pixel coordinates and the pixel coordinates in the first monocular image.

[0165] In some embodiments of the present specification, the motion trajectory of an object can be better estimated through pixel coordinate information in multiple frames of images; by predicting the pixel coordinates of the target object in a third monocular image and comparing them with the actual pixel coordinates, the accuracy of the motion estimation can be verified and the accuracy of the determined initial depth information can be further improved, which is conducive to improving the accuracy of positioning.

[0166] In some embodiments, the processor may optimize the first pixel coordinates in the third monocular image based on the actual pixel coordinates in the third monocular image to determine the second pixel coordinates of the target object in the third monocular image; and calculate the initial depth information of the target object based on the second pixel coordinates in the third monocular image and the pixel coordinates in the first monocular image, as well as the camera calibration parameters.

[0167] The actual pixel coordinates refer to the actual pixel coordinates of the target object in the third monocular image.

[0168] In some embodiments, the processor may obtain the actual pixel coordinates in a variety of ways. For example, the processor may obtain the third image through an image acquisition device, input the third image into the target detection model, obtain the pixel coordinates of the target object in the third image, and determine them as the actual pixel coordinates. Exemplarily, when the first image is a left-eye image at a first moment, the processor may obtain an image taken by a right-eye camera at the first moment as the third image.

[0169] The second pixel coordinates are estimated pixel coordinates of the optimized target object in the third monocular image.

[0170] In some embodiments, the processor may correct the first pixel coordinates in the third monocular image through an optimization algorithm so that the difference between the first pixel coordinates in the third monocular image and the actual pixel coordinates in the third monocular image is minimized, and the first pixel coordinate corresponding to the minimum difference is used as the second pixel coordinate in the third monocular image.

[0171] The optimization algorithm may include but is not limited to the least squares method, the gradient descent method, etc.

[0172] In some embodiments, the processor may determine a second disparity based on the second pixel coordinates in the third monocular image and the pixel coordinates in the first monocular image; and calculate depth information of the target object based on the second disparity and camera calibration parameters (such as focal length and baseline distance, etc.).

[0173] The second disparity reflects the disparity of the target object in the first monocular image and the second monocular image.

[0174] In some embodiments of the present specification, the motion trajectory of the target object is estimated by the pixel coordinates in multiple frames of images, and the accuracy of depth estimation is improved by predicting the pixel coordinates and comparing them with the actual pixel coordinates, thereby improving the accuracy of the initial depth information estimation, which is beneficial for subsequent target ranging and positioning.

[0175] In some embodiments, the processor can obtain multiple chessboard images of a preset chessboard at different angles and orientations; determine the pixel coordinates of the corner points in each chessboard image; determine the projection coordinates of the corner points in the world coordinate system; and determine the camera calibration parameters based on the projection coordinates and the pixel coordinates.

[0176] The camera calibration parameters are used to describe the internal and external characteristics of the camera, as well as the distortion of the lens. For example, the camera calibration parameters may include the camera's intrinsic parameter matrix, extrinsic parameter matrix, and the camera's distortion parameters.

[0177] In some embodiments, the processor may calibrate the binocular camera using a variety of methods, such as Zhang Zhengyou's chessboard calibration method.

[0178] A chessboard image refers to an image containing a chessboard.

[0179] Corner points are the intersections of the chessboard.

[0180] The pixel coordinates of the corner points refer to the positions of the corner points in the coordinate system corresponding to the chessboard image.

[0181] The projection coordinates of the corner point in the world coordinate system refer to the actual physical coordinates of the corner point in the world coordinate system.

[0182] Exemplarily, the calibration process includes: preparing a calibration plate with a checkerboard pattern, the size of the calibration plate and the size of the grid are known. For example, the number of grids on each side of the checkerboard is m×n, and the side length of each grid is s millimeters. Use a camera to shoot the calibration plate from different angles and distances to obtain a set of checkerboard images.

[0183] In some embodiments, the processor may use a corner detection algorithm (such as Harris corner detection or findChessboardCorners function in OpenCV) to detect the corner positions of the calibration plate in each chessboard image and determine the pixel coordinates of the corners in the chessboard image.

[0184] In some embodiments, the processor can calculate the physical coordinates of each corner point in the world coordinate system based on a preset world coordinate system (eg, the first corner point in the upper left corner of the chessboard as the origin) and according to the known chessboard size and grid size.

[0185] In some embodiments, the processor may solve the intrinsic and extrinsic parameter matrices by minimizing a re-projection error, ie, the difference between physical coordinates and pixel coordinates.

[0186] In some embodiments of the present specification, through camera calibration, the intrinsic and extrinsic parameters of the binocular camera can be accurately obtained, providing necessary parameters for subsequent binocular vision tasks (such as depth calculation, three-dimensional coordinates, etc.).

[0187] Step S230 , based on the initial depth information, intercepting a target area from the binocular image of the target object.

[0188] A binocular image refers to a set of images containing a target object captured by an image acquisition device. For example, a binocular image may include a left-eye image and a right-eye image of a target object captured by a binocular camera at the same time (hereinafter referred to as a left-eye image and a right-eye image of a target object).

[0189] In some embodiments, the processor may communicate with an image acquisition device (e.g., a binocular camera) to continuously, periodically, or intermittently acquire binocular images sent by the image acquisition device. In some embodiments, the binocular camera may upload the captured binocular images to a storage device, and the processor may communicate with the storage device to acquire the binocular images from the storage device in real time or at intervals.

[0190] The target area refers to the area where the target object is located in the binocular image.

[0191] In some embodiments, the target area may include a first area and a second area.

[0192] The first area refers to an area in the target left-eye image corresponding to the initial depth information; the second area refers to an area in the target right-eye image corresponding to the initial depth information.

[0193] In some embodiments, the processor can determine the depth information corresponding to each pixel in the target left-eye image based on the target left-eye image in a manner similar to that described above, and determine the target depth range based on the initial depth information; based on the target depth range, intercept the area in the target left-eye image where the depth information is within the target depth range as the first area.

[0194] In some embodiments, the processor can determine the depth information corresponding to each pixel in the target right eye image based on the target right eye image in a manner similar to that described above, and determine the target depth range based on the initial depth information; based on the target depth range, intercept the area in the target right eye image where the depth information is within the target depth range as the second area.

[0195] The target depth range is used to filter the range of depth values ​​of the area where the target object is located.

[0196] Step S240: determining target depth information of the target object based on the target area.

[0197] Target depth information refers to the result of fine positioning of the distance between the target object and the image acquisition device (such as a binocular camera, etc.).

[0198] In some embodiments, the processor can determine the pixel coordinates of the target object in the first area and the pixel coordinates of the second area respectively through target recognition based on the first area and the second area, and determine the target depth information of the target object based on the pixel coordinates of the first area and the pixel coordinates of the second area, as well as the camera calibration parameters.

[0199] In some embodiments, the processor may input the target area into the target detection model to obtain the pixel coordinates of the target object in the target area; and determine the target depth information of the target object based on the pixel coordinates of the target object in the target area.

[0200] In some embodiments, the processor can determine the target depth information of the target object in a variety of ways based on the pixel coordinates of the target object in the target area. For example, the processor can compare the first area with the calibration image, and determine the first target depth information of the target object according to a pre-calibrated correspondence relationship; compare the second area captured by the binocular camera with the calibration image, and determine the second target depth information of the target object according to a pre-calibrated correspondence relationship; average the first target depth information and the second target depth information (e.g., arithmetic average, weighted average, etc.) to determine the target depth information of the target object. For more information about calibrated images, please refer to Figure 2The relevant description above.

[0201] In some embodiments of the present specification, by determining the depth information of the target object through the target area, the amount of calculation can be reduced, the image area required for target detection can be reduced, and the amount of calculation can be reduced.

[0202] In some embodiments, the processor may determine target depth information of the target object based on pixel coordinates of the target object in the target area and camera calibration parameters.

[0203] In some embodiments, the processor can input the first area and the second area into the target detection model respectively to obtain detection results; extract the coordinates and category information of each detection box from the detection results, and the processor can filter out the detection box coordinates of the target object based on the category index of the target object (for example, the category index of the AGV vehicle is 0), and determine the pixel coordinates of the target object in the first area and the pixel coordinates in the second area.

[0204] In some embodiments, the processor can obtain the pixel coordinates (U) of the target object in the first area from the target detection model. L , V L ) and the pixel coordinates in the first region (U R , V R ); based on pixel coordinates (U L , V L )Pixel coordinates (U R , V R ) Calculate the disparity (U L -U R ); calculating the depth information of the target object based on the parallax, focal length and baseline distance; calculating the target depth information of the target object based on the depth information of the target object and the intrinsic parameter matrix and the extrinsic parameter matrix.

[0205] In some embodiments, the processor may determine the disparity of the target object in the target area based on the pixel coordinates of the target area; calculate the product of the focal length and the center distance of the camera calibration parameters, and determine the target depth information of the target object based on the ratio of the product to the disparity.

[0206] Exemplarily, the target depth information can be calculated by formula (1):

[0207]

[0208] Among them, Z is the target depth information, f is the focal length, and B is the baseline distance.

[0209] In some embodiments of the present specification, the accuracy of positioning and measurement can be improved by using the triangulation positioning principle of a binocular camera.

[0210] In some embodiments of the present specification, a binocular camera can be used to accurately calculate the distance between the target object and the camera, which is more accurate than using only the two-dimensional information provided by a monocular camera or a lidar or other sensor, and helps the vehicle to perceive the surrounding environment more accurately. Based on an improved target detection model (such as YOLO, etc.), the target detection task can be completed efficiently and accurately, and combined with the real-time image input of the binocular camera, real-time target ranging and positioning can be achieved.

[0211] For areas such as workshops and factories, there may be multiple target objects such as AGVs running therein. However, AGVs of the same model have the same appearance, and it is difficult to distinguish between the various AGVs through images.

[0212] In some embodiments, a unique identifier can be set on or around the AGV, and the target vehicle to be located (e.g., the vehicle that sends the location request) can be determined through the identifier, so that the AGV can be easily identified in the field of view of the binocular camera. For example, a QR code or RFID tag can be installed on the AGV, and the binocular camera can identify the target vehicle by scanning the identifier.

[0213] It should be noted that by using the image information of the monocular camera to complete the rough positioning of the target during idle time, and then using the binocular ranging principle and target detection model to perform precise measurement when precise positioning is needed, the system energy consumption can be saved while achieving efficient and accurate target positioning.

[0214] In some embodiments, process 200 further includes: if the target object includes a target vehicle and a target cargo, performing driving planning for the target vehicle based on the target depth information of the target vehicle and the target depth information of the target cargo.

[0215] Target cargo refers to the cargo that needs to be transported.

[0216] Driving planning includes route control and vehicle control; route control includes lane change, route planning / adjustment, etc.; vehicle control includes vehicle acceleration, vehicle acceleration, vehicle turning, vehicle U-turn, starting vehicle, stopping vehicle, etc.

[0217] Exemplarily, after the processor obtains a binocular image of the surrounding environment by means of a binocular camera set around the vehicle (such as directly in front), it determines the target depth information of the AGV and the target depth information of the cargo based on the binocular image, and then determines the relative position information (such as direction, distance, etc.) of the AGV and the cargo, and automatically controls the AGV vehicle to go to the designated cargo area to complete the corresponding handling and unloading operations.

[0218] In some embodiments of the present specification, the reliability and safety of vehicle driving planning can be improved by efficiently and accurately locating the target vehicle and the target cargo.

[0219] In some embodiments, process 200 further includes: if the target object includes a target vehicle and an obstacle, performing driving planning for the target vehicle based on target depth information of the target vehicle and target depth information of the obstacle.

[0220] Obstacles are objects that block the travel of the target vehicle.

[0221] In some embodiments, the processor may control the target vehicle to avoid the obstacle based on the target depth information of the target vehicle and the target depth information of the obstacle.

[0222] It should be noted that the above description of the process 200 is only for example and illustration, and does not limit the scope of application of this specification. For those skilled in the art, various modifications and changes can be made to the process 200 under the guidance of this specification. However, these modifications and changes are still within the scope of this specification.

[0223] Figure 3 This is an exemplary flow chart of determining a detection object according to some embodiments of this specification.

[0224] In some embodiments, the target detection model includes a first network, a second network, and a third network connected in sequence. Before the detection object is recognized based on the initial monocular image, a pre-trained initial model needs to be obtained to recognize the detection image using the pre-trained initial model. The initial model is a target detection model trained using multiple training samples.

[0225] In some embodiments, the processor may determine the detected object through process 300 based on the target detection model.

[0226] In some embodiments, process 300 may be implemented by a processor based on a ranging positioning system. Figure 3 As shown, process 300 includes the following steps:

[0227] Step S310, using the optimization module of the first network to extract features of the detection object in the initial monocular image to obtain feature information of multiple different scales.

[0228] The first network is used to extract features of different scales from the input initial monocular image.

[0229] Feature information may include, but is not limited to, any one or combination of edge features, texture features, color features, shape features, local structures, etc.

[0230] The optimization module is used to extract features from input images of different sizes.

[0231] In some embodiments, the backbone network includes multiple levels, and each level may include an optimization module.

[0232] In some embodiments, the input initial monocular image can be downsampled in sequence through multiple sequentially connected optimization modules to obtain feature information of multiple different scales. For more information about optimization, please refer to Figure 4 Related description.

[0233] Step S320, using the attention mechanism of the second network to determine the weight of each feature information, and based on the weight, perform feature enhancement on multiple feature information to obtain multiple enhanced features; based on the multiple enhanced features, obtain multiple fusion features.

[0234] The second network is used to fuse features of different scales and output fused features of different scales.

[0235] The attention mechanism dynamically adjusts the weight of each feature information by increasing the weight of important parts (such as the size, shape, and outline of the object) and reducing the weight of unimportant parts (such as irrelevant background information) when processing information.

[0236] Enhanced features refer to feature maps processed by the attention mechanism.

[0237] Fusion features refer to the combination of multiple feature information to form a comprehensive feature representation.

[0238] In some embodiments, the attention mechanism may include a channel attention mechanism, a spatial attention mechanism, etc. The channel weight may be obtained through the channel attention mechanism, and the spatial weight may be obtained through the spatial attention mechanism.

[0239] In some embodiments, the feature map can be adjusted through a channel attention mechanism and a spatial attention mechanism, and the adjusted feature map can be subjected to feature fusion to obtain a fused feature.

[0240] In some embodiments, the attention mechanism is a grouped attention module, which determines the weights of the input feature information in multiple channels based on the grouped attention module, and determines the multiple channel attention features of the feature information by weighting the weights of the multiple channels; and performs feature fusion and residual connection on the multiple channel attention features to obtain enhanced features. For more information about the grouped attention module, please refer to Figure 4 Related description.

[0241] Step S330, using the third network to process multiple fusion features, determine whether there is a detection object in the initial monocular image, and if there is a detection object in the initial monocular image, mark the detection box of the detection object in the initial monocular image.

[0242] The third network performs classification regression prediction on fusion features of different scales through different detection modules to obtain detection results.

[0243] In some embodiments, the third network is used to perform target detection and classification tasks on the fused features. For example, the third network can predict a classification score (called HeatMap) at each pixel of the fused features, and HeatMap represents the probability that each pixel in the fused features is the center of the target object. The value of each pixel reflects the probability that the position is the center of the target object. The larger the HeatMap value, the higher the probability that the position is the center of the target object.

[0244] In some embodiments, the third network may predict the offset of the center point of each potential target object, and locate the specific position of the center of the target object based on the offset.

[0245] In some embodiments, the third network may predict the detection box size (width and height) of the center point of each potential target object, as well as other regression parameters, to generate a final target object detection box.

[0246] In some embodiments of the present specification, in the second network, the feature pyramid part is changed from PAN (Path Aggregation Network) to PAFPN (Pyramid Attention Feature Pyramid Network), which can obtain more subtle information and solve the problem of premature loss of image detail information during the downsampling process. The addition of an attention mechanism can improve the performance of the target detection model, maintain high accuracy of target detection when the number of parameters is extremely small, and obtain a more real-time effect.

[0247] In some embodiments, the target detection model is trained through the following steps: inputting the training sample into the initial model to obtain the predicted detection box of the training sample; determining the loss value of the initial model based on the predicted detection box and the label detection box of the training sample; training the initial model based on the loss value to obtain the target detection model.

[0248] In some embodiments, the training is performed based on a large number of labeled training samples through various feasible methods. For example, parameter updating can be performed based on the gradient descent method. An exemplary training process includes: inputting a plurality of labeled training samples into the initial model, constructing a loss function through the label detection box and the predicted detection box of the initial model, and iteratively updating the parameters of the initial effect prediction model through gradient descent or other methods based on the loss value of the loss function. When the preset conditions are met, the model training is completed and a trained effect prediction model is obtained. Among them, the preset conditions can be that the loss function converges, the number of iterations reaches a threshold, etc.

[0249] In some embodiments, the first training sample includes at least sample images, and the sample images may include two categories: one category of images is sample images labeled with objects of different categories, and the other category is sample images without objects. The training sample may be obtained based on historical data.

[0250] In some embodiments, the label detection frame may include the actual category and the actual detection frame corresponding to the sample image. The label detection frame may be obtained by a processor or manual annotation.

[0251] In some embodiments, the loss function of the target detection model is a Generalized Focal Loss loss function.

[0252] In some embodiments of the present specification, by optimizing modules and attention mechanisms, the target detection model can better capture the key features of the target object and improve the accuracy and robustness of detection; by constructing a lightweight convolutional neural network and attention mechanism, the model can better capture the key features of the target object, meet the real-time processing needs of mobile applications, edge devices, IoT devices, etc. with limited computing power and storage space, improve the generalization ability of the model, and improve the performance and efficiency of target detection tasks.

[0253] In some embodiments, the target detection model can be obtained based on improvements to the Nanodet model.

[0254] Figure 4 is an exemplary schematic diagram of a target detection model according to some embodiments of the present specification.

[0255] In some embodiments, the initial monocular image is processed by a trained target detection model to obtain a detected object, which specifically includes the following steps:

[0256] Step S410: Input the initial monocular image into the initial model, perform multi-scale convolution processing through the first network, and obtain the first feature, the second feature and the third feature of the initial monocular image.

[0257] like Figure 4As shown, the first network includes a first feature layer, a second feature layer and a third feature layer connected in sequence, and the first feature layer, the second feature layer and the third feature layer all include an optimization module. Figure 4 As shown, the first feature layer includes three optimization modules connected in sequence. The second feature layer includes an optimization module, and the third feature layer includes an optimization module.

[0258] In some embodiments, feature extraction is performed on an initial monocular image through a first network, including: inputting the initial monocular image into an initial model, performing 8-fold downsampling and feature extraction on the initial monocular image through a first feature layer of the first network, and obtaining a first feature output by the first feature layer; performing 2-fold downsampling and feature extraction on the first feature output by the first feature layer through a second feature layer, and obtaining a second feature output by the second feature layer; performing 2-fold downsampling and feature extraction on the second feature output by the second feature layer through a third feature layer, and obtaining a third feature output by the third feature layer, so as to obtain feature information of multiple scales.

[0259] In some embodiments, the input initial monocular image is sequentially downsampled through multiple sequentially connected optimization modules to obtain feature information of multiple different scales, including: the first feature layer, the second feature layer and the third feature layer all include optimization modules, wherein the feature processing in the first feature layer includes: the initial monocular image is downsampled by 2 and feature extraction is performed by the first optimization module of the first feature layer, the output of the first optimization module is downsampled by 2 and feature extraction is performed by the second optimization module, and the output of the second optimization module is downsampled by 2 and feature extraction is performed by the third optimization module, to obtain the first feature of 8 times downsampling. The feature processing in the second feature layer includes: the output of the first feature layer is downsampled by 2 and feature extraction is performed by the features of the optimization module, to obtain the second feature of 16 times downsampling. The feature processing in the third feature layer includes: the output of the second feature layer is downsampled by 2 and feature extraction is performed by the features of the optimization module, to obtain the third feature of 32 times downsampling.

[0260] For more information about the optimization module, see Figure 5 Related description.

[0261] In some embodiments, the optimization module may include at least one convolution module.

[0262] The convolution module is used to perform convolution operations on the input image to extract local features of the image.

[0263] In some embodiments, the convolution module may include at least one or any combination of a convolution kernel, a normalization function, an activation function, etc. For example, the convolution module may be a CBS (Convolution-Batch Normalization-SiLU) module, wherein the size of the convolution kernel may be 3×3, the normalization function may be a BN (Batch Normalization) function, and the activation function may be a relu (Rectified Linear Unit) function, a Sigmoid function, etc.

[0264] In some embodiments, multiple feature layers of the first network include optimization modules, which can perform multi-scale feature extraction, improve feature diversity and improve the real-time performance of feature extraction, and provide a solid foundation for subsequent multi-scale-based multiple feature predictions.

[0265] Figure 5 is an exemplary schematic diagram of an optimization module according to some embodiments of the present specification.

[0266] like Figure 5 As shown, in some embodiments, the optimization module includes a first convolution module, a segmentation layer, a bottleneck layer, a splicing layer, and a second convolution module, wherein the input end of the first convolution module is the input end of the optimization module, the output end of the first convolution module is connected to the input end of the segmentation layer, and the first output end of the segmentation layer is connected to the input end of the bottleneck layer; the bottleneck layer extracts features through a plurality of serially connected Bottleneck layers; the second output end of the segmentation layer, the output end of the bottleneck layer, and the output end of each Bottleneck layer are respectively connected to the input end of the splicing layer, and spliced ​​through the splicing layer to obtain a spliced ​​feature map; the spliced ​​feature map is processed by the second convolution module to obtain the output of the optimization module.

[0267] The input end of the first convolution module is the input end of the optimization module. The CBS module is used to perform convolution, batch normalization (BatchNorm) and activation function on the input feature map. The first convolution module can be used to initialize feature extraction, such as extracting basic features through the convolution layer, and stabilizing the training process of the input feature map through the batch normalization layer. The activation function is used to increase the nonlinear characteristics of the input feature map.

[0268] The split layer is used to split the feature map output by the first convolution module into two feature maps, one feature map is passed to the bottleneck layer, and the other feature map is used as the input of the subsequent splicing layer.

[0269] The bottleneck layer includes one or more serially connected Bottleneck layers to extract deeper features. In a lightweight network, the bottleneck module can include a 1×1 convolutional layer (for feature dimensionality reduction) and a 3×3 convolutional layer (for feature extraction).

[0270] The concatenation layer is used to concatenate the output of the segmentation layer, the output of the last bottleneck layer, and the output of each intermediate bottleneck layer to obtain a concatenated feature map.

[0271] Multi-level feature information is integrated by concatenating some feature maps retained by the segmentation layer with the feature maps processed by all bottleneck modules.

[0272] The second convolution module is used to further fuse and process the concatenated feature maps output by the concatenation layer. For example, the second convolution module can adjust the size and number of channels of the concatenated feature maps through the convolution layer, and increase the expressiveness of the concatenated feature maps through the batch normalization layer and the activation function.

[0273] It should be noted that “first”, “second”, etc. are only used for the convenience of distinction. For example, the structures of the first convolution module and the second convolution module can be exactly the same, and the same applies to others.

[0274] In some embodiments of the present specification, feature fusion and enhancement of the optimization module are used to help achieve target detection tasks in a lightweight deep learning network.

[0275] Step S420: The first feature, the second feature, and the third feature are fused through the second network to obtain a first fused feature, a second fused feature, and a third fused feature.

[0276] In some embodiments, the second network of the traditional Nanodet model includes an upsampling structure and a downsampling structure, and a grouping attention module can be added between the upsampling structure and the downsampling structure in the second network of the traditional Nanodet model. For example, at least two of the first feature, the second feature, and the third feature can be upsampled and concatenated to obtain three initial features, and the three initial features are input into the first fusion layer, the second fusion layer, and the third fusion layer respectively. The first fusion layer, the second fusion layer, and the third fusion layer all include a grouping attention module to perform feature fusion processing respectively to obtain a first fusion feature output by the first fusion layer, a second fusion feature output by the second fusion layer, and a third fusion feature output by the third fusion layer.

[0277] Step S430: predict the first fusion feature, the second fusion feature and the third fusion feature respectively through the third network to obtain the detected object.

[0278] After obtaining the first fusion feature, the second fusion feature and the third fusion feature, the first fusion feature, the second fusion feature and the third fusion feature are respectively identified and predicted through the third network to obtain the identification data output by the third network, that is, the detected object.

[0279] In some embodiments, the third network includes a first detection module, a second detection module, and a third detection module; wherein,

[0280] The first detection module, the second detection module and the third detection module of the head network process the first fusion feature, the second fusion feature and the third fusion feature respectively, that is, the first detection module 1 detects and identifies the first fusion feature, the second detection module 2 detects and identifies the second fusion feature, and the third detection module 3 detects and identifies the third fusion feature, thereby obtaining the detection data of the first detection module, the second detection module and the third detection module, and then performing non-maximum suppression on the three detection data based on the non-maximum suppression technology, thereby obtaining the optimal detection result output, and obtaining the final category information and detection frame of the detected object.

[0281] In this embodiment, through the second network, multi-scale fusion features can be more accurately fused, thereby more accurately identifying the detected object, providing an effective data basis for the subsequent coordinate positioning of the detected object.

[0282] In some embodiments, the second network includes an initial fusion layer, and a first fusion layer, a second fusion layer, and a third fusion layer connected in sequence. Specifically, in step S420, the first feature, the second feature, and the third feature are fused by the second network, including:

[0283] S421: Fusing at least two of the first feature, the second feature, the third feature, etc. through an initial fusion layer to obtain a first initial feature, a second initial feature, and a third initial feature.

[0284] The initial fusion layer includes multiple upsampling modules and multiple concatenation modules. The first fusion layer, the second fusion layer, and the third fusion layer all include multiple downsampling modules and multiple grouping attention modules.

[0285] The processing of the initial fusion layer includes: upsampling the third feature output by the third feature layer through the first upsampling module; and using the third feature output by the third feature layer as the third initial feature;

[0286] The output feature map of the first up-sampling module is superimposed on the second feature output by the second feature layer through the first superposition module to output a second initial feature.

[0287] The second upsampling module is used to upsample the output feature map of the first superposition module;

[0288] The second superposition module is used to superimpose the output feature map of the second upsampling module with the first feature map output by the first feature layer to output the first initial feature.

[0289] S422: The first initial feature is fused by the first fusion layer, the second initial feature is fused by the second fusion layer, and the third initial feature is fused by the third fusion layer to obtain the first fused feature, the second fused feature and the third fused feature.

[0290] The first fusion layer, the second fusion layer, and the third fusion layer all include a group attention module.

[0291] The processing of the first fusion layer includes: processing the first initial feature through the first grouping attention module and outputting a first fusion feature;

[0292] The first downsampling module is used to downsample the output feature map of the first group attention module.

[0293] The processing of the second fusion layer includes: processing the second initial feature through a second grouping attention module;

[0294] The output feature map of the first downsampling module and the output feature map of the second grouping attention module are superimposed by a third superposition module to output a second fusion feature;

[0295] Downsampling the output feature map of the third stacking module by the second downsampling module;

[0296] The processing of the third fusion layer includes: processing the third initial feature through a third grouping attention module;

[0297] The output feature map of the second downsampling module is superimposed on the output feature map of the first grouping attention module through the fourth superposition module to output a third fusion feature.

[0298] In some embodiments, upsampling may be achieved by interpolation (eg, bilinear interpolation) or deconvolution.

[0299] In some embodiments, downsampling can be achieved by a convolution operation with a stride of 2 or pooling (such as maximum pooling or average pooling).

[0300] In some embodiments, after 5 downsampling operations in the first network, 2 times, 4 times, 8 times, 16 times and 32 times downsampled feature maps are obtained respectively, and the 8 times, 16 times and 32 times downsampled feature maps are input into the second network for multi-scale feature fusion. The second network includes an upsampling structure and a downsampling structure, and the upsampling structure is used to: upsample the 32 times downsampled feature map by bilinear interpolation, and directly add it to the 16 times downsampled feature map to obtain a fused 16 times feature map; then the fused 16 times feature map is further upsampled by bilinear interpolation, and directly added and fused with the 8 times downsampled feature map, and the 8 times fused result is output. The downsampling structure is used to: downsample the fused 8-fold feature map (i.e., the output of the first group attention module) through linear interpolation, add and fuse it with the fused 16-fold feature map (i.e., the output of the second group attention module), and output a 16-fold downsampled fusion result; then downsample the 16-fold downsampled fusion result through linear interpolation, add and fuse it with the fused 32-fold downsampled feature map (i.e., the output of the third group attention module), and output a 32-fold downsampled fusion result.

[0301] It should be noted that "first", "second", etc. are only used for convenient distinction. For example, the structures of the first upsampling module and the second upsampling module can be exactly the same, and the same is true for other things. For example, the structures of the first superposition module and the second superposition module can be the same.

[0302] In some embodiments of the present specification, a wide-scale attention module is added between the upsampling structure and the downsampling structure to filter out features lost during the linear interpolation process, extract the area that the network pays most attention to, and reduce model loss.

[0303] Figure 6 It is an exemplary schematic diagram of the grouping attention module shown in some embodiments of the present specification.

[0304] like Figure 6 As shown, in some embodiments, the group attention module includes multiple convolutional layers, a channel attention module, multiple weight layers, and a residual splicing module; wherein,

[0305] In some embodiments, the weights of input feature information in multiple channels are determined based on a grouped attention module, including: performing a segmentation operation on a feature map input to the grouped attention module to obtain a feature map of a first branch and a feature map of a second branch; performing different processing on the feature map of the first branch and the feature map of the second branch respectively to obtain a processed feature map of the first branch and a processed feature map of the second branch; based on the processed feature map of the first branch and the processed feature map of the second branch, extracting weights through a channel attention module, and outputting channel weights of the first branch and channel weights of the second branch.

[0306] Among them, the feature map of the input group attention module is segmented to obtain the feature map of the first branch and the feature map of the second branch, including: inputting the feature map of the input group attention module into a convolutional layer (e.g., 1×1Conv), and performing a segmentation operation on the result of the convolutional layer to obtain a first feature map of the first branch and a second feature map of the second branch.

[0307] The step of performing different processing on the feature map of the first branch and the feature map of the second branch to obtain the processed feature map of the first branch and the processed feature map of the second branch includes:

[0308] Input the first feature map into a convolutional layer (e.g., 3×3Conv) to obtain a third feature map;

[0309] Input the second feature map into a convolutional layer (e.g., 3×3Conv) to obtain a fourth feature map;

[0310] Superimposing the third characteristic map and the fourth characteristic map to obtain a fifth characteristic map;

[0311] The fifth feature map is input into a convolutional layer (e.g., 3×3Conv) to obtain a sixth feature map.

[0312] The third feature map and the sixth feature map are the feature map of the processed first branch and the feature map of the processed second branch, respectively.

[0313] Wherein, based on the processed feature map of the first branch and the processed feature map of the second branch, the weights are extracted through the channel attention module, and the channel weights of the first branch and the channel weights of the second branch are output, including: based on the third feature map and the sixth feature map, the channel weights of the first branch and the channel weights of the second branch are obtained through the channel attention module; for obtaining the channel weights of the first branch and the channel weights of the second branch, please refer to Figure 6 The relevant description is as follows.

[0314] By adopting two branches to determine the channel weights of different branches, the number of parameters can be reduced.

[0315] In some embodiments, the weights of the multiple channels are weighted to determine the multiple channel attention features of the feature information, including: performing weighted calculation on the third feature map and the channel weight of the first branch to obtain the seventh feature map, and performing weighted calculation on the sixth feature map and the channel weight of the second branch to obtain the eighth feature map;

[0316] The seventh feature map and the eighth feature map are used as multiple channel attention features of the two branches.

[0317] In some embodiments, feature fusion and residual connection are performed on multiple channel attention features to obtain enhanced features, including:

[0318] Input the seventh feature map and the eighth feature map into a convolutional layer (e.g., 1×1Conv) to obtain a ninth feature map;

[0319] The ninth feature map and the original feature map are input into the residual concatenation module to obtain the output of the group attention module as the enhanced feature.

[0320] Among them, the convolution layer can perform convolution operations to capture the features of the input image. It should be noted that the convolution layer is the basic building block of the convolution module, and the convolution module is a modular structure that can be composed of multiple convolution layers and other components.

[0321] The segmentation operation is used to segment the output of the eighth convolutional layer to obtain two feature maps after segmentation, namely the first feature map of the first branch and the second feature map of the second branch. The two branches will be processed through different paths respectively.

[0322] The superposition operation is used to superimpose the third feature map and the fourth feature map to obtain a fifth feature map.

[0323] The channel attention module is used to calculate the third feature map and the sixth feature map to obtain the channel weight of the first branch and the channel weight of the second branch.

[0324] The first weight layer uses the channel weight of the first branch to perform weighted calculation on the third feature map to obtain the seventh feature map; the second weight layer uses the channel weight of the second branch to perform weighted calculation on the sixth feature map to obtain the eighth feature map.

[0325] The residual splicing module is used to superimpose the ninth feature map with the original input feature map to obtain the final output feature map.

[0326] In some embodiments of the present specification, adding a group attention module can obtain more subtle information and solve the problem of premature loss of image detail information during the downsampling process, thereby improving the target detection performance and maintaining high precision and real-time performance of target detection when the parameter amount is extremely small.

[0327] In some embodiments, the channel attention module includes a pooling layer, multiple convolutional layers, and an activation layer, and the channel attention module is used to calculate the importance of different feature maps.

[0328] In some embodiments, the processed feature map of the first branch and the processed feature map of the second branch can be superimposed to obtain a fused feature map; the fused feature map can be pooled to obtain global information of multiple channels; based on the global information of multiple channels, the channel weight of the first branch and the channel weight of the second branch can be determined.

[0329] Among them, the processed feature map of the first branch and the processed feature map of the second branch are superimposed to obtain a fused feature map, including: the channel attention module can superimpose the third feature map and the sixth feature map (such as splicing based on the channel dimension, etc.) to obtain a fused feature map.

[0330] The fused feature map is pooled to obtain global information of multiple channels, including: the fused feature map is pooled by global average of the pooling layer to obtain global information of multiple channels.

[0331] For example, the fused feature map can be input into a pooling formula to obtain global information of multiple channels. The pooling formula is formula (3):

[0332]

[0333] Among them, S c is the global information of the cth channel, is the fused feature map, H and W are the height and width of the fused feature map respectively, and C is the number of channels.

[0334] By extracting global information and avoiding the loss of feature information, it helps to obtain the key information of the feature map.

[0335] In some embodiments, based on the global information of multiple channels, the channel weight of the first branch and the channel weight of the second branch are determined, including: passing the global information of multiple channels through convolution operations of two convolutional layers in sequence to obtain local information of the first branch and local information of the second branch; normalizing the local information of the first branch and the local information of the second branch to obtain the channel weight of the first branch and the channel weight of the second branch.

[0336] The global information of multiple channels is sequentially passed through the convolution operation of two convolutional layers to obtain the local information of the first branch and the local information of the second branch, including: the global information of multiple channels is sequentially passed through the convolution operation of the first convolutional layer and the convolution operation of the second convolutional layer to obtain the local information of the first branch and the local information of the second branch.

[0337] Among them, the local information of the first branch and the local information of the second branch are normalized to obtain the channel weight of the first branch and the channel weight of the second branch, including: the local information of the first branch and the local information of the second branch are activated by an activation layer to obtain the channel weight of the first branch and the channel weight of the second branch.

[0338] Exemplarily, the activation process may be implemented based on a normalization process formula, and the normalization process formula is formula (5):

[0339]

[0340] in, is the local information of the i-th branch in the c-th channel, i = 1, 2; a i (c) is the channel weight of the i-th branch in the c-th channel.

[0341] Through the channel weights of the two branches, splitting the weighted combination can help capture the key information in the feature map.

[0342] In some embodiments, the first convolution layer may include a 1x1 Conv and a normalization layer (e.g., BatchNorm) and an activation function layer (e.g., ReLU function) connected in sequence, and the second convolution layer may be a 1x1 Conv.

[0343] In some embodiments, the first pooling layer is a global pooling layer (eg, Global Pooling).

[0344] In some embodiments, the activation layer may employ a SoftMax function.

[0345] In some embodiments of the present specification, the connection and path cascade of the feature pyramid network are enhanced through the attention mechanism, providing a more powerful multi-scale feature representation, so that the network can better capture complex semantic information. The feature pyramid can handle the multi-scale variation problem in target recognition without increasing the computational overhead; the extracted features can generate multi-scale feature representations, including high-resolution features.

[0346] Figure 7 FIG. 1 is an exemplary schematic diagram of another grouping attention module according to some embodiments of the present specification. Figure 7 As shown, in some embodiments, the group attention module uses two branches (N=2) to reduce the number of parameters. The two branches include Split1 and Split2. Both branches include a 1x1 Conv and a 3x3 Conv. In Split2, the fourth feature map output by the tenth convolutional layer is summed with the third feature map output by the ninth convolutional layer in Split1, and the summed result is input into the eleventh convolutional layer (e.g., 3x3 Conv), which can receive semantic information from the two split groups and expand the network's receptive field. The sum of the third feature map and the fourth feature map (the middle sum) can be determined by formula (2):

[0347]

[0348] in, is the sum of the third and fourth feature maps, X i Represents the input image information, F iRepresented as the result information of the output feature map of Split(i), i=1,2.

[0349] The channel statistics generated by the first pooling layer collect global spatial information as formula (3):

[0350]

[0351] α and β represent the point-by-point spatial information of the feature map in the height and width directions, respectively.

[0352] The channel attention mechanism is used to aggregate the weighted fusion represented by the basis array, where the split weighted combination can capture the key information in the feature map. The feature map of the cth channel is calculated as formula (4):

[0353]

[0354] a i Assign a (soft) weight to i It can be determined by formula (5):

[0355]

[0356] in, Represents the weight of the global spatial information S on the cth channel, which is quantized by two 1x1Conv (BatchNorm and ReLU activation).

[0357] In some embodiments, a standard residual architecture (ResNet) is used, and the ninth feature map is summed with the original feature map input into the residual concatenation module to obtain the output of the grouped attention module. Figure 7 In , + indicates the concatenation of channel dimensions, such as c' / 2+c' / 2=c'. X indicates weighted calculation of input dimensional information or spatial information to extract important information.

[0358] In some embodiments of the present specification, in the second network, upsampling and downsampling are completed by linear interpolation, and multi-scale feature maps are directly added to complete the fusion, which can reduce the amount of calculation, but feature information will be lost. A grouping attention module is added between the upsampling result and the downsampling result to screen the features lost in the linear interpolation process, extract the area that the network pays most attention to, and reduce model loss; the grouping attention module helps to enhance the connection and path cascading of the feature pyramid network.

[0359] Figure 8 It is an exemplary module diagram of a ranging and positioning system according to some embodiments of this specification.

[0360] like Figure 8As shown, one or more embodiments of this specification further provide a ranging and positioning system 800, the system comprising:

[0361] The detection module 810 is used to process the collected initial monocular image through the trained target detection model to obtain the detected object;

[0362] An acquisition module 820, configured to acquire initial depth information of a target object if there is a target object in the detected object;

[0363] A capture module 830, configured to capture a target area from the binocular image of the target object based on the initial depth information;

[0364] The positioning module 840 is used to determine the target depth information of the target object based on the target area.

[0365] The specific implementation of each of the above modules can be found in the previous embodiments and will not be described in detail here.

[0366] Fig. 9 is a schematic diagram of the structure of an electronic device according to some embodiments of this specification. Fig. 9 As shown, the electronic device 900 may include: a processor 901, a memory 902. The electronic device 900 may also include one or more of a multimedia component 903, an input / output (I / O) component 904, and a communication component 905. In this embodiment, the electronic device 900 may be a device integrated in a vehicle to interact with the aforementioned ranging-based positioning system to implement the ranging positioning method provided in this embodiment. It should be understood that the electronic device 900 may also include some components in the aforementioned ranging-based positioning system.

[0367] The processor 901 is used to control the overall operation of the electronic device 900 to complete all or part of the steps in the above-mentioned ranging and positioning method. The memory 902 is used to store various types of data to support the operation of the auxiliary electronic device 900. These data may include instructions for any application or method used to operate on the auxiliary electronic device 900, as well as application-related data, such as contact data, messages sent and received, pictures, audio, video, etc. The memory 902 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random Access Memory, SRAM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read Only Memory, EEPROM), erasable programmable read-only memory (Erasable Programmable Read Only Memory, EPROM), programmable read-only memory (Programmable Read Only Memory, PROM), read-only memory (Read Only Memory, ROM), magnetic memory, flash memory, magnetic disk or optical disk. The multimedia component 903 may include a screen and an audio component. The screen may be, for example, a touch screen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signal may be further stored in the memory 902 or sent through the communication component 905. The audio component also includes at least one speaker for outputting audio signals. The I / O component 904 provides an interface between the processor 901 and other interface modules, and the above-mentioned other interface modules may be keyboards, mice, buttons, etc. These buttons may be virtual buttons or physical buttons. The communication component 905 is used for wired or wireless communication between the electronic device 900 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, 4G, Narrow Band Internet of Things (NB-IOT), Enhanced Narrow Band Internet of Things technology (Enhanced Machine-Type Communication, eMTC), or other 5G, etc., or a combination of one or more of them, is not limited here. Therefore, the corresponding communication component 905 may include: Wi-Fi module, Bluetooth module, NFC module, etc.

[0368] In an exemplary embodiment, the electronic device 900 can be implemented by one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), controllers, microcontrollers, microprocessors or other electronic components to perform the above-mentioned ranging and positioning method.

[0369] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, and when the program instructions are executed by a processor, the steps of the above-mentioned ranging and positioning method are implemented. For example, the computer-readable storage medium may be the above-mentioned memory 902 including program instructions, and the above-mentioned program instructions may be executed by the processor 901 of the electronic device 900 to complete the steps of the ranging and positioning method provided in any of the above-mentioned embodiments;

[0370] Alternatively, when the instructions are executed by a computer, the instructions are executed to complete the steps of the ranging and positioning method provided in any of the above embodiments.

[0371] The present application also provides a vehicle, on which is provided a ranging-based positioning system provided in any one of the above embodiments, and the ranging-based positioning system is used to execute the ranging-based positioning method provided in any one of the above embodiments.

[0372] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0373] The embodiments, implementation methods and related technical features of the present application can be combined and replaced with each other without conflict.

[0374] The above are only preferred embodiments of the present application and do not constitute any form of limitation to the present application. Although the descriptions of various embodiments in the embodiments of the present application have different focuses, for parts not described in detail in a certain embodiment, reference can be made to the relevant embodiments of other embodiments. However, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present application without departing from the content of the technical solution of the present application are still within the scope of the technical solution of the present application.

Claims

1. A ranging and positioning method, characterized in that: The method comprises: The collected initial monocular image is processed by the trained target detection model to obtain the detected object; If there is a target object in the detected object, obtaining initial depth information of the target object; Based on the initial depth information, intercepting a target area from the binocular image of the target object; Based on the target area, target depth information of the target object is determined.

2. The method according to claim 1, characterized in that The obtaining of initial depth information of the target object comprises: Respectively acquiring a first monocular image and a second monocular image of the target object at different times; Initial depth information of the target object is acquired based on the first monocular image and the second monocular image.

3. The method according to claim 2, characterized in that The acquiring initial depth information of the target object based on the first monocular image and the second monocular image includes: Inputting the first monocular image and the second monocular image into the target detection model respectively, obtaining the pixel coordinates of the target object in the first monocular image and the pixel coordinates of the target object in the second monocular image; Determine a motion vector of the target object based on pixel coordinates in the first monocular image and pixel coordinates in the second monocular image; Initial depth information of the target object is acquired based on the motion vector and pixel coordinates in the first monocular image.

4. The method according to claim 3, characterized in that The acquiring initial depth information of the target object based on the motion vector and the pixel coordinates in the first monocular image includes: Predicting, based on the motion vector and the pixel coordinates in the first monocular image, a first pixel coordinate of the target object in a third monocular image, wherein the third monocular image is an image of a different perspective taken at the same time point as the first monocular image; Initial depth information of the target object is acquired based on first pixel coordinates in the third monocular image and pixel coordinates in the first monocular image.

5. The method according to claim 4, characterized in that The acquiring initial depth information of the target object based on the first pixel coordinates in the third monocular image and the pixel coordinates in the first monocular image includes: Optimizing the first pixel coordinates in the third monocular image based on the actual pixel coordinates in the third monocular image to determine the second pixel coordinates of the target object in the third monocular image; Initial depth information of the target object is calculated based on the second pixel coordinates in the third monocular image and the pixel coordinates in the first monocular image, and camera calibration parameters.

6. The method according to claim 1, characterized in that The determining, based on the target area, target depth information of the target object includes: Inputting the target area into a target detection model to obtain pixel coordinates of the target object in the target area; Based on the pixel coordinates of the target object in the target area, target depth information of the target object is determined.

7. The method according to claim 1, characterized in that The determining the target depth information of the target object based on the pixel coordinates of the target object in the target area includes: Based on the pixel coordinates of the target object in the target area and the camera calibration parameters, target depth information of the target object is determined.

8. The method according to claim 7, characterized in that The determining the target depth information of the target object based on the pixel coordinates of the target object in the target area and the camera calibration parameters includes: Determining the disparity of the target object in the target area based on the pixel coordinates of the target area; The product of the focal length and the center distance of the camera calibration parameters is calculated, and the target depth information of the target object is determined based on the ratio of the product to the parallax.

9. The method according to claim 1, characterized in that: After determining the target depth information of the target object based on the target area, the method further includes: If the target object includes a target vehicle and a target cargo, driving planning is performed on the target vehicle based on the target depth information of the target vehicle and the target depth information of the target cargo.

10. The method according to claim 1, characterized in that The method further comprises: Acquire multiple chessboard images of preset chessboard grids at different angles and orientations; Determine the pixel coordinates of each corner point in the chessboard image; Determine the projection coordinates of the corner point in the world coordinate system; A camera calibration parameter is determined based on the projection coordinates and the pixel coordinates.

11. The method according to claim 1, characterized in that The target detection model includes a first network, a second network and a third network connected in sequence; The method processes the collected initial monocular image by using the trained target detection model to obtain the detected object, including: Using the optimization module of the first network to extract features of the target object in the initial monocular image, to obtain feature information of multiple scales; Determine the weight of each of the feature information by using the attention mechanism of the second network, and enhance the features of the plurality of feature information based on the weights to obtain a plurality of enhanced features; and obtain a plurality of fused features based on the plurality of enhanced features; The third network is used to process the multiple fusion features, determine whether the detection object exists in the initial monocular image, and determine whether the detection object exists in the initial monocular image, and mark the detection frame of the detection object in the initial monocular image.

12. The method according to claim 11, characterized in that The using the optimization module of the first network to extract features of the target object in the initial monocular image to obtain feature information of multiple scales, including: The input initial monocular image is downsampled in sequence through a plurality of sequentially connected optimization modules to obtain the feature information of the plurality of different scales.

13. The method according to claim 11, characterized in that The attention mechanism is a grouped attention module. The using the attention mechanism of the second network to determine the weight of each of the feature information, and enhancing the multiple feature information based on the weights to obtain multiple enhanced features, including: Based on the grouped attention module, the weights of the input feature information in multiple channels are determined, and the weights of the multiple channels are weighted to determine multiple channel attention features of the feature information; feature fusion and residual connection are performed on the multiple channel attention features to obtain the enhanced features.

14. The method according to claim 13, characterized in that The step of determining the weights of the input feature information in multiple channels based on the grouped attention module includes: Perform a segmentation operation on the feature map of the input group attention module to obtain a feature map of the first branch and a feature map of the second branch; Performing different processing on the feature graph of the first branch and the feature graph of the second branch respectively to obtain a processed feature graph of the first branch and a processed feature graph of the second branch; Based on the processed feature map of the first branch and the processed feature map of the second branch, weights are extracted through a channel attention module, and the channel weights of the first branch and the channel weights of the second branch are output.

15. The method according to claim 14, characterized in that The step of extracting weights through a channel attention module based on the processed feature map of the first branch and the processed feature map of the second branch, and outputting the channel weights of the first branch and the channel weights of the second branch includes: Superimposing the processed feature map of the first branch and the processed feature map of the second branch to obtain a fused feature map; Performing a pooling operation on the fused feature map to obtain global information of multiple channels; Based on the global information of the multiple channels, a channel weight of the first branch and a channel weight of the second branch are determined.

16. The method according to claim 15, characterized in that The pooling operation is performed on the fused feature map to obtain global information of multiple channels, including: The fused feature map is input into the pooling formula to obtain the global information of the multiple channels. The pooling formula is: Among them, S c is the global information of the cth channel, is the fused feature map, H and W are the height and width of the fused feature map respectively, and C is the number of channels.

17. The method according to claim 15, characterized in that The determining, based on the global information of the plurality of channels, the channel weight of the first branch and the channel weight of the second branch comprises: The global information of the multiple channels is sequentially subjected to convolution operations of two convolutional layers to obtain local information of the first branch and local information of the second branch; The local information of the first branch and the local information of the second branch are normalized to obtain the channel weight of the first branch and the channel weight of the second branch. The formula for the normalization is: in, is the local information of the i-th branch in the c-th channel, i = 1, 2; a i (c) is the channel weight of the i-th branch in the c-th channel.

18. A ranging and positioning system, characterized in that: The system comprises: The detection module is used to process the collected initial monocular image through the trained target detection model to obtain the detected object; An acquisition module, configured to acquire initial depth information of a target object if there is a target object in the detected object; A capture module, configured to capture a target area from the binocular image of the target object based on the initial depth information; A positioning module is used to determine target depth information of the target object based on the target area.

19. A computer-readable storage medium having a computer program stored thereon, characterized in that: The computer-readable storage medium stores instructions, which, when executed by a computer, enable the computer to implement the ranging and positioning method according to any one of claims 1 to 17.

20. An electronic device, characterized in that: include: a memory having a computer program stored thereon; A processor, configured to execute the computer program in the memory to implement the ranging and positioning method according to any one of claims 1 to 17.

21. A vehicle, characterized in that: Including the ranging and positioning system described in claim 18.

Images