Object positioning method and system, storage medium, electronic equipment and vehicle

Abstract

The invention relates to an object positioning method and system, a storage medium, electronic equipment and a vehicle, and the method comprises the steps: carrying out the processing of a collected monocular image through a trained target detection model, and obtaining a detection object and a target vehicle in the monocular image; if the detected objects comprise the target object, obtaining a first binocular image of the target object and a second binocular image of the target vehicle; and determining relative azimuth information of the target object relative to the target vehicle based on the first binocular image and the second binocular image. According to the method, more accurate distance detection and positioning are realized through the improved target detection model, and the method is suitable for being deployed on edge equipment.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and in particular to an object positioning method, system, storage medium, electronic device and vehicle. Background Art

[0002] When currently AGVs (Automated Guided Vehicles) are used in factories and warehouses, although they have certain autonomous navigation capabilities, during the operation of the AGV, there may be defects or deficiencies in the internal data processing logic, or changes in the external environment, such as the sudden appearance of obstacles, changes in light conditions, wear and tear of ground markings, or other unforeseen factors, which may cause the AGV to lose its path tracking ability and be unable to continue to perform tasks according to the scheduled route, and manual intervention is required to recalibrate or troubleshoot. Therefore, it is necessary to accurately locate the AGV to solve the navigation loss problem that may be encountered during its operation. Summary of the invention

[0003] The embodiments of the present application provide an object positioning method, system, storage medium, electronic device and vehicle to solve the positioning problem of AGV.

[0004] In order to achieve the above object, according to a first aspect of the present application, an object positioning system is provided, and the method includes:

[0005] The collected monocular image is processed by a trained target detection model to obtain the detection object and the target vehicle in the monocular image;

[0006] If the detected objects include a target object, acquiring a first binocular image of the target object and a second binocular image of the target vehicle;

[0007] Based on the first binocular image and the second binocular image, relative position information of the target object relative to the target vehicle is determined.

[0008] Optionally, the target detection model is trained by the following steps: inputting a training sample into an initial model to obtain a predicted detection box of the training sample;

[0009] Determining a loss value of the initial model according to the predicted detection frame and the label detection frame of the training sample;

[0010] The initial model is trained according to the loss value to obtain the target detection model.

[0011] Optionally, determining the loss value of the initial model according to the predicted detection box and the label detection box of the training sample includes:

[0012] Determining an initial intersection-over-union ratio based on the predicted detection frame and the label detection frame;

[0013] Based on the magnitude relationship between the initial intersection-and-union ratio and a preset threshold, the initial intersection-and-union ratio is normalized to obtain a target intersection-and-union ratio;

[0014] Based on the target intersection-over-union ratio, the loss value is determined.

[0015] Optionally, the normalizing the initial intersection-and-union ratio based on the size relationship between the initial intersection-and-union ratio and a preset threshold to obtain a target intersection-and-union ratio includes:

[0016] If the initial intersection-over-union ratio is less than a first threshold, setting the target intersection-over-union ratio to a first preset value;

[0017] If the initial intersection-over-union ratio is less than a second threshold, setting the target intersection-over-union ratio to a second preset value;

[0018] If the initial intersection-over-union ratio is within the range of the first threshold and the second threshold, the initial intersection-over-union ratio is normalized to obtain the target intersection-over-union ratio.

[0019] Optionally, the target detection model includes a first network, a second network and a third network connected in sequence;

[0020] The method of processing the collected monocular image by the trained target detection model to obtain the detection object and the target vehicle in the monocular image includes:

[0021] Using the optimized convolution module of the first network to extract features of the target object in the monocular image, and obtain feature information of multiple scales;

[0022] Determine the weight of each of the feature information by using the attention mechanism of the second network, and enhance the features of the plurality of feature information based on the weights to obtain a plurality of enhanced features; and obtain a plurality of fused features based on the plurality of enhanced features;

[0023] The third network is used to process the multiple fusion features to determine whether the detection object and the target vehicle exist in the monocular image, and when the detection object and / or the target vehicle exist in the monocular image, a detection frame of the detection object and / or the target vehicle is marked in the monocular image.

[0024] Optionally, the using the optimized convolution module of the first network to extract features of the target object in the monocular image to obtain feature information of multiple scales includes:

[0025] Based on at least one deformable convolution module in the optimized convolution module, feature information of the monocular image is extracted by adjusting the size and position of the convolution kernel.

[0026] Optionally, the attention mechanism is a group attention module, which uses the attention mechanism of the second network to determine the weight of each of the feature information, and enhances the multiple feature information based on the weight to obtain multiple enhanced features, including:

[0027] Based on the grouped attention module, the weights of the input feature information in multiple channels are determined, and the weights of the multiple channels are weighted to determine multiple channel attention features of the feature information; feature fusion and residual connection are performed on the multiple channel attention features to obtain the enhanced features.

[0028] Optionally, determining the weights of the input feature information in multiple channels based on the grouped attention module includes:

[0029] Perform a segmentation operation on the feature map of the input group attention module to obtain a feature map of the first branch and a feature map of the second branch;

[0030] Performing different processing on the feature graph of the first branch and the feature graph of the second branch respectively to obtain a processed feature graph of the first branch and a processed feature graph of the second branch;

[0031] Based on the processed feature map of the first branch and the processed feature map of the second branch, weights are extracted through a channel attention module, and the channel weights of the first branch and the channel weights of the second branch are output.

[0032] Optionally, extracting weights through a channel attention module based on the processed feature map of the first branch and the processed feature map of the second branch, and outputting the channel weight of the first branch and the channel weight of the second branch include:

[0033] Superimposing the processed feature map of the first branch and the processed feature map of the second branch to obtain a fused feature map;

[0034] Performing a pooling operation on the fused feature map to obtain global information of multiple channels;

[0035] Based on the global information of the multiple channels, a channel weight of the first branch and a channel weight of the second branch are determined.

[0036] Optionally, performing a pooling operation on the fused feature map to obtain global information of multiple channels includes:

[0037] The fused feature map is input into the pooling formula to obtain the global information of the multiple channels. The pooling formula is:

[0038]

[0039] Among them, S c is the global information of the cth channel, is the fused feature map, H and W are the height and width of the fused feature map respectively, and C is the number of channels.

[0040] Optionally, determining the channel weight of the first branch and the channel weight of the second branch based on the global information of the multiple channels includes:

[0041] The global information of the multiple channels is sequentially subjected to convolution operations of two convolutional layers to obtain local information of the first branch and local information of the second branch;

[0042] The local information of the first branch and the local information of the second branch are normalized to obtain the channel weight of the first branch and the channel weight of the second branch. The formula for the normalization is:

[0043]

[0044] in, is the local information of the i-th branch in the c-th channel, i = 1, 2; a i (c) is the channel weight of the i-th branch in the c-th channel.

[0045] Optionally, determining the relative position information of the target object relative to the target vehicle based on the first binocular image and the second binocular image includes:

[0046] Inputting the first binocular image and the second binocular image into the target detection model respectively, obtaining the pixel coordinates of the target object in the first binocular image and the pixel coordinates of the target vehicle in the second binocular image;

[0047] Based on the pixel coordinates in the first binocular image and the pixel coordinates in the second binocular image, relative orientation information of the target object relative to the target vehicle is determined.

[0048] Optionally, the relative position information of the target object relative to the target vehicle is determined based on the pixel coordinates in the first binocular image and the pixel coordinates in the second binocular image:

[0049] Determining depth information of the target object based on pixel coordinates of the target object in the first binocular image and camera calibration parameters;

[0050] Determining depth information of the target vehicle based on pixel coordinates of the target vehicle in the second binocular image and the camera calibration parameters;

[0051] Based on the depth information of the target object and the depth information of the target vehicle, relative position information of the target object with respect to the target vehicle is determined.

[0052] Optionally, determining the depth information of the target object based on the pixel coordinates of the target object in the first binocular image and camera calibration parameters includes:

[0053] Determining a disparity of the target object in the first binocular image based on pixel coordinates of the target object in the first binocular image;

[0054] The product of the focal length and the center distance of the camera calibration parameters is calculated, and the depth information of the target object is determined based on the ratio of the product to the parallax.

[0055] Optionally, the method further comprises:

[0056] Acquire multiple chessboard images of preset chessboard grids at different angles and orientations;

[0057] Determine the pixel coordinates of each corner point in the chessboard image;

[0058] Determine the projection coordinates of the corner point in the world coordinate system;

[0059] The camera calibration parameters are determined based on the projection coordinates and the pixel coordinates.

[0060] Optionally, after determining the relative position information of the target object relative to the target vehicle based on the first binocular image and the second binocular image, the method further includes:

[0061] If the target object is an obstacle, controlling the target vehicle to bypass the target object according to the relative position information;

[0062] If the target object is cargo, the target vehicle is controlled to travel to the target object according to the relative position information.

[0063] According to a second aspect of the present application, an embodiment of the present application further provides an object positioning system, the system comprising:

[0064] A first detection module is used to process the collected monocular image through a trained target detection model to obtain a detection object and a target vehicle in the monocular image;

[0065] A second detection module, configured to obtain a first binocular image of the target object and a second binocular image of the target vehicle if the detected objects include the target object;

[0066] A positioning module is used to determine the relative position information of the target object relative to the target vehicle based on the first binocular image and the second binocular image.

[0067] According to the third aspect of the present application, an embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored, and the computer-readable storage medium stores instructions, which, when executed by a computer, enable the computer to implement any object positioning method provided in the embodiment of the present application.

[0068] According to a fourth aspect of the present application, an embodiment of the present application further provides an electronic device, including:

[0069] a memory having a computer program stored thereon;

[0070] A processor is used to execute the computer program in the memory to implement any object positioning method provided in the embodiments of the present application.

[0071] According to a fifth aspect of the present application, an embodiment of the present application also provides a vehicle, comprising the object positioning system.

[0072] Some embodiments of the present specification include at least the following beneficial effects: during idle time, only the monocular camera is started to detect target objects in the surrounding environment. If the target object is detected, the binocular camera is used to measure the distance to complete the precise distance measurement, thereby completing the precise positioning, which can achieve more accurate distance detection and positioning. This method has the advantages of low cost, fast processing speed, and lightweight model, and is suitable for deployment on edge devices.

[0073] Other features and advantages of the present application will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application, and those skilled in the art can obtain other drawings based on these drawings without creative work.

[0075] In order to more completely understand the present application and its beneficial effects, the following description will be given in conjunction with the accompanying drawings, wherein the same figure numbers represent the same parts in the following description.

[0076] Figure 1 is an application scenario diagram of an object positioning system according to some embodiments of this specification;

[0077] Figure 2 is an exemplary flow chart of an object positioning method according to some embodiments of this specification;

[0078] Figure 3 is an exemplary flow chart for determining a detection object and a target vehicle according to some embodiments of this specification;

[0079] Figure 4 is an exemplary schematic diagram of a target detection model according to some embodiments of this specification;

[0080] Figure 5 is an exemplary schematic diagram of an optimized convolution module according to some embodiments of this specification;

[0081] Figure 6 is an exemplary schematic diagram of a grouping attention module according to some embodiments of the present specification;

[0082] Figure 7 is an exemplary schematic diagram of another grouping attention module according to some embodiments of the present specification;

[0083] Figure 8 is an exemplary structural diagram of an object positioning system according to some embodiments of this specification;

[0084] Fig. 9 It is a schematic diagram of the structure of an electronic device according to some embodiments of this specification. DETAILED DESCRIPTION

[0085] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0086] In order to facilitate understanding of the implementation scheme provided in the embodiment of the present application, the relevant application background of the object positioning system provided in the embodiment of the present application is first described.

[0087] At present, in order to solve the positioning problem of AGV, a distance measurement and positioning method based on a depth camera is proposed to locate AGV. Due to the use of a depth camera, the cost is relatively high; and the detection and segmentation model takes a long time to process point cloud data, which is prone to jamming when deployed on edge devices. In addition, based on a binocular camera, traditional image processing algorithms (such as Canny edge detection, template matching, etc.) or deep learning-based target detection algorithms (such as YOLO) can be used for target detection and positioning, but traditional image processing algorithms are easily affected by the environment and similar objects, and template matching takes a long time, which affects the accuracy of distance measurement and positioning; and the YOLOv8 model still faces the problem of feature information being easily lost in the deep network and insufficient feature fusion capabilities in the field of small target detection, making it difficult to ensure the accuracy and efficiency of small target detection.

[0088] In view of this, some embodiments of this specification provide an object positioning system, which processes the video stream of the binocular camera based on the improved YOLOV8 model deployment, realizes real-time target tracking and distance measurement of AGV, and improves the accuracy of positioning.

[0089] The system and method of the present application can be applied to various scenarios of automated environments that require precise positioning and navigation, such as industrial manufacturing, logistics warehousing, and healthcare. By detecting and locating specific raw material stacking points or finished product stacking points, AGV can accurately complete the task of transporting materials.

[0090] It should be noted that the description order of the following embodiments is not intended to limit the preferred order of the embodiments. Although the logical order is shown in the flow chart, in some cases, the steps shown or described may be performed in an order different from that shown in the drawings.

[0091] Figure 1 It is an application scenario diagram of the object positioning method shown in some embodiments of this specification.

[0092] like Figure 1 As shown, the application scenario 100 of the object positioning system may include a processor 110 , a network 120 , a vehicle 130 , and an image acquisition device 140 .

[0093] The processor 110 may be a local physical server, a cloud server (such as Figure 1 as shown), a virtual server, a distributed server, or any other suitable computing device.

[0094] In some embodiments, the processor 110 may be integrated into the image acquisition device 140 .

[0095] In some embodiments, the processor 110 may acquire data of the surrounding environment. The processor 110 may use the acquired data to determine the position information of various objects in the surrounding environment. For example, the processor 110 may obtain data from the image acquisition device 140. The processor 110 may communicate with the image acquisition device 140 and / or other components of the vehicle 130 via the network 120.

[0096] The network 120 may include but is not limited to any one or combination of wireless local area network (WLAN), wide area network (WAN), wireless network of radio wave / cellular network / satellite communication network and / or local wireless network or short-range wireless network (Bluetooth™).

[0097] In some embodiments, the processor 110 can process the collected monocular image through the trained target detection model to obtain the detection object and the target vehicle in the monocular image; if the detection object includes the target object, a first binocular image of the target object and a second binocular image of the target vehicle are obtained; based on the first binocular image and the second binocular image, the relative position information of the target object relative to the target vehicle is determined. For more information about this embodiment, please refer to Figure 2 Related description.

[0098] The vehicle 130 may be a transportation service vehicle for the application scenario 100. In some embodiments, the vehicle 130 may include a taxi, a private car, a ride-sharing car, a shared vehicle, a bus bicycle, a tricycle, a motorcycle, a robot, an AGV, etc., or any combination thereof. In some embodiments, the vehicle 130 may include a private self-driving car, a shared self-driving car, an unmanned automatic freight car, etc. In some embodiments, the vehicle 130 may receive instructions issued by the processor 110 and complete corresponding tasks in accordance with the instructions. For example, the vehicle 130 may receive instructions such as performing an autonomous driving operation and automatically control the vehicle 130 to complete the corresponding operation.

[0099] The image acquisition device 140 is used to capture data of the surrounding environment. In some embodiments, the image acquisition device 140 includes but is not limited to cameras, webcams, video cameras, other devices with camera functions (mobile phones, tablets, etc.), etc. The camera may include a monocular camera, a binocular camera, a depth camera, etc. The camera may also be two cameras (such as a first monocular camera, a second monocular camera, etc.) with an optical system installed side by side and aligned to point in the same direction. Since the binocular camera has two viewpoints, the image taken by the binocular camera contains depth information. It is conceivable that other suitable cameras or sensors capable of sensing depth information may also be used. The image acquisition device 140 can record a video consisting of multiple image frames captured at multiple time points. In some embodiments, the image acquisition device 140 can capture a series of binocular image frames of a scene. The binocular image frames can be sent to the processor 110 in real time (e.g., by streaming), or sent to the processor 110 together after the vehicle 130 completes a portion of the trajectory.

[0100] In some embodiments, the image acquisition device 140 can be a monitoring device in the environment, and the image acquisition device 140 can also be installed on the target environment based on a mounting structure. In some embodiments, the mounting structure can use screws, adhesives or other mounting structures. In some embodiments, any suitable mounting structure can also be used.

[0101] In some embodiments, the image acquisition device 140 can be deployed at least one location in the target environment. In some embodiments, the number and deployment location of the image acquisition device 140 can be preset. For example, the target environment can be divided to obtain multiple areas of the target environment, and each area can be corresponding to the installation of the image acquisition device 140. The target environment can be the environment where the image acquisition device is located, such as a factory, a warehouse, or a logistics park. In some embodiments, the image acquisition device 140 can communicate with one or more components (such as the vehicle 130, the processor 110) in the application scenario through the network 120.

[0102] It should be noted that application scenario 100 is provided only for the purpose of illustration and is not intended to limit the scope of this specification. For those of ordinary skill in the art, various modifications or changes can be made according to the description of this specification. For example, application scenario 100 can also include a database, an information source, etc. For another example, application scenario 100 can be implemented on other devices to achieve similar or different functions. However, changes and modifications will not deviate from the scope of this specification.

[0103] Figure 2 is an exemplary flow chart of an object positioning method according to some embodiments of the present specification.

[0104] In some embodiments, process 200 may be implemented by a processor of an object positioning system. Figure 2 As shown, process 200 includes the following steps:

[0105] Step S210, processing the collected monocular image by using the trained target detection model to obtain the detection object and the target vehicle in the monocular image.

[0106] The target prediction model is a model for obtaining the location and / or category information of an object. In some embodiments, the target prediction model may be a machine learning model, which may include but is not limited to any one or combination of a convolutional neural network (CNN) model, a neural network (NN) model, or other custom model structures.

[0107] In some embodiments, the input of the target prediction model includes an image to be detected, and the output is a detection result, which may include the detection box coordinates and category information of the detected object in the image to be detected.

[0108] In some embodiments, the target prediction model can be trained by a large number of training data sets with training labels. In some embodiments, the training can be performed based on the training data sets by various methods. For example, the training can be performed based on the gradient descent method.

[0109] In some embodiments, the target prediction model can also be obtained based on other network forms and corresponding network training. For example, the training of the target prediction model can be implemented by the YOLOv8 model, and the YOLOv8 model can be configured to be obtained by training based on the training data set. For specific model content, please refer to Figure 3 The image to be detected may be an image captured by an image acquisition device. For example, the image to be detected may include a monocular image, a left image in a first binocular image, a right image in a first binocular image, a left image in a second binocular image, a right image in a second binocular image, etc. For more information about the first binocular image and the second binocular image, please refer to Figure 2 The relevant description is as follows.

[0110] A monocular image refers to an image of the surrounding environment captured by a single camera in the environment.

[0111] In some embodiments, the monocular image may be an image captured by a monocular camera (eg, a first monocular camera).

[0112] In some embodiments, the monocular image may also be an image taken by a camera in a binocular camera. For example, the monocular image may be a left-eye image or a right-eye image.

[0113] For the purpose of easy distinction, the two cameras in the binocular camera are taken as a left camera and a right camera as examples.

[0114] The left-eye image is the image captured by the left-eye camera under the preset field of view.

[0115] The right-eye image is the image captured by the right-eye camera under the preset field of view.

[0116] The preset field of view is the range of view that the camera can capture under specific settings.

[0117] In some embodiments, the left-eye image and the right-eye image are images taken by the left-eye camera and the right-eye camera respectively at the same time or a similar time.

[0118] The detected object refers to the specific object that is identified in the monocular image. For example, the detected object may include other vehicles, goods, obstacles, people, etc.

[0119] The target vehicle refers to a vehicle that needs to be located. For example, a vehicle may send a location request to the processor, and the processor may determine the vehicle as a target vehicle based on the location request.

[0120] In some embodiments, the processor can be connected to the image acquisition device for communication to obtain a monocular image of the target object. For example, during idle time (e.g., after the AGV completes a handling task and waits for the next task), the processor can control one of the binocular cameras to turn on (e.g., the left camera), and the processor can be connected to the camera for communication to continuously or periodically obtain a monocular image (or video stream) taken by a single camera, and input it into the target detection model for processing to obtain a detection result, which may include the detection frame and category information of each detection object, and / or the detection frame and category information of the target vehicle.

[0121] It should be noted that by using the image information of the monocular camera to complete the rough positioning of the target during idle time, and then using the binocular ranging principle and target detection model to perform precise measurement when precise positioning is needed, the system energy consumption can be saved while achieving efficient and accurate target positioning.

[0122] For areas such as workshops and factories, there may be multiple target objects such as AGVs running therein. However, AGVs of the same model have the same appearance, and it is difficult to distinguish between the various AGVs through images.

[0123] In some embodiments, a unique identifier can be set on or around the AGV, and the target vehicle to be located (e.g., the vehicle that sends the location request) can be determined through the identifier, so that the AGV can be easily identified in the field of view of the binocular camera. For example, a QR code or RFID tag can be installed on the AGV, and the binocular camera can identify the target vehicle by scanning the identifier.

[0124] In some embodiments, the processor can process the same frame of monocular image based on the target detection model to identify multiple different objects, including detection objects and target vehicles. For example, in a monocular image containing a warehouse environment, the target detection model can identify detection objects, target vehicles, etc.

[0125] In some embodiments, the processor may process two different monocular image frames based on the target detection model to identify multiple different objects contained in each monocular image, such as a detection object and / or a target vehicle. The two different monocular image frames may be two frames separated by a preset number of frames or a preset time.

[0126] The preset number of frames or the preset time may be determined based on experiments or experience.

[0127] Step S220: If the detected objects include the target object, a first binocular image of the target object and a second binocular image of the target vehicle are acquired.

[0128] The target object refers to a specific object that needs to be identified and located. For example, the target object may include goods to be transported, obstacles, etc.

[0129] In some embodiments, the processor can determine whether there is a target object of the target category in the monocular image based on the detection results output by the target detection model. If there is a target object, the binocular camera or another monocular camera (such as a second monocular camera) is turned on; if there is no target object, the single camera continues to work.

[0130] The target category of detected objects refers to objects that can help determine the location of the vehicle or have a direct impact on the operation of the vehicle. For example, the target category of detected objects may include obstacles, cargo, etc.

[0131] In some embodiments, the processor may be connected in communication with the image acquisition device to obtain a first binocular image of the target object and a second binocular image of the target vehicle. For example, the processor may issue a control instruction to control the binocular camera to turn on another camera while detecting the target object. The processor may be connected in communication with the binocular camera to continuously or periodically obtain a first binocular image of the target object and a second binocular image of the target vehicle taken by the binocular camera.

[0132] The first binocular image refers to a binocular image containing a target object.

[0133] The second binocular image refers to a binocular image containing the target vehicle.

[0134] In some embodiments, the first binocular image may include a first left-eye image taken by a left-eye camera and a first right-eye image taken by a right-eye camera; the second binocular image may include a second left-eye image taken by the left-eye camera and a second right-eye image taken by the right-eye camera.

[0135] In some embodiments, the first binocular image and the second binocular image may be two groups of images taken by the binocular camera at different times. The first binocular image and the second binocular image may also be the same group of images taken by the binocular camera.

[0136] Step S230: determining relative position information of the target object relative to the target vehicle based on the first binocular image and the second binocular image.

[0137] The relative position information refers to the position information of the target object relative to the target vehicle. For example, the relative position information may include distance, direction, etc.

[0138] In some embodiments, the relative position information of the target object relative to the target vehicle can be determined in a variety of ways based on the first binocular image and the second binocular image. For example, the relative position information of the target object relative to the target vehicle can be determined by manual analysis, theoretical calculation and / or modeling.

[0139] In some embodiments, the processor may input the first binocular image and the second binocular image into the target detection model respectively to obtain the pixel coordinates of the target object in the first binocular image and the pixel coordinates of the target vehicle in the second binocular image; based on the pixel coordinates in the first binocular image and the pixel coordinates in the second binocular image, determine the relative orientation information of the target object relative to the target vehicle. The pixel coordinates are the coordinates of the object in the pixel coordinate system corresponding to the image.

[0140] In some embodiments, the target detection model may output the pixel coordinates of the four vertices of the detection box (e.g., the detection box of the target object or the detection box of the target vehicle) as pixel coordinates. In some embodiments, the pixel coordinates may also be the coordinates of the center point of the detection box, or the coordinates of the upper left corner or the lower right corner of the detection box.

[0141] In some embodiments, when identifying a target object and / or a target vehicle in an image to be detected, the image to be detected needs to be preprocessed, wherein the preprocessing includes grayscale conversion, median filtering, contour search, etc. on the image to be detected, and then identifying the target object in the image to be detected or the image frame stream.

[0142] In some embodiments, the target detection model can process the first left-eye image of the first binocular image and output the pixel coordinates of the target object in the first left-eye image; the target detection model can process the first right-eye image of the first binocular image and output the pixel coordinates of the target object in the first right-eye image.

[0143] In some embodiments, the target detection model can process the second left image of the second binocular image and output the pixel coordinates of the target vehicle in the second left image; the target detection model can process the second right image of the second binocular image and output the pixel coordinates of the target vehicle in the second right image.

[0144] In some embodiments of the present specification, the pixel coordinates in the first binocular image and the pixel coordinates in the second binocular image are used to facilitate the subsequent determination of the three-dimensional coordinates of the target object and the three-dimensional coordinates of the target vehicle, which is beneficial for subsequent precise positioning, navigation and obstacle avoidance.

[0145] In some embodiments, the processor can determine the depth information of the target object based on the pixel coordinates of the target object in the first binocular image and the camera calibration parameters; determine the depth information of the target vehicle based on the pixel coordinates of the target vehicle in the second binocular image and the camera calibration parameters; determine the relative orientation information of the target object relative to the target vehicle based on the depth information of the target object and the depth information of the target vehicle.

[0146] Depth information refers to the distance information of an object in three-dimensional space. For example, the depth information may include the actual distance and direction from a camera to a certain point on the surface of the object.

[0147] The camera calibration parameters are used to describe the internal and external characteristics of the camera, as well as the distortion of the lens. For example, the camera calibration parameters may include the camera's intrinsic parameter matrix, extrinsic parameter matrix, and the camera's distortion parameters.

[0148] In some embodiments, the processor may calibrate the binocular camera using a variety of methods, such as Zhang Zhengyou's chessboard calibration method.

[0149] In some embodiments, the processor can obtain multiple chessboard images of a preset chessboard at different angles and orientations; determine the pixel coordinates of the corner points in each chessboard image; determine the projection coordinates of the corner points in the world coordinate system; and determine the camera calibration parameters based on the projection coordinates and the pixel coordinates.

[0150] A chessboard image refers to an image containing a chessboard.

[0151] Corner points are the intersections of the chessboard.

[0152] The pixel coordinates of the corner points refer to the positions of the corner points in the coordinate system corresponding to the chessboard image.

[0153] The projection coordinates of the corner point in the world coordinate system refer to the actual physical coordinates of the corner point in the world coordinate system.

[0154] Exemplarily, the calibration process includes: preparing a calibration plate with a checkerboard pattern, the size of the calibration plate and the size of the grid are known. For example, the number of grids on each side of the checkerboard is m×n, and the side length of each grid is s millimeters. Use a camera to shoot the calibration plate from different angles and distances to obtain a set of checkerboard images.

[0155] In some embodiments, the processor may use a corner detection algorithm (such as Harris corner detection or findChessboardCorners function in OpenCV) to detect the corner positions of the calibration plate in each chessboard image and determine the pixel coordinates of the corners in the chessboard image.

[0156] In some embodiments, the processor can calculate the physical coordinates of each corner point in the world coordinate system based on a preset world coordinate system (eg, the first corner point in the upper left corner of the chessboard as the origin) and according to the known chessboard size and grid size.

[0157] In some embodiments, the processor may solve the intrinsic and extrinsic parameter matrices by minimizing a re-projection error, ie, the difference between physical coordinates and pixel coordinates.

[0158] In some embodiments of the present specification, through camera calibration, the intrinsic and extrinsic parameters of the binocular camera can be accurately obtained, providing necessary parameters for subsequent binocular vision tasks (such as depth calculation, three-dimensional coordinates, etc.).

[0159] In some embodiments, the processor may determine the depth information of the target object in a variety of ways based on the pixel coordinates of the target object in the first binocular image and the camera calibration parameters. In some embodiments, the processor may obtain the pixel coordinates of the target object in the first left eye image (U L , V L ) and the pixel coordinates in the first right image (U R , V R ); based on pixel coordinates (U L , V L )Pixel coordinates (U R , V R ) Calculate the disparity (U L -U R ) ; Calculate the depth information of the target object based on parallax, focal length, and baseline distance.

[0160] For example, the processor may determine the disparity of the target object in the first binocular image based on the pixel coordinates of the target object in the first binocular image; calculate the product of the focal length and the center distance of the camera calibration parameters, and determine the depth information of the target object based on the ratio of the product to the disparity. Exemplarily, the target depth information may be calculated by formula (1).

[0161] In some embodiments, the depth information may be calculated using formula (1):

[0162]

[0163] Among them, Z is the depth information, f is the focal length, and B is the baseline distance.

[0164] In some embodiments, the processor may determine the depth information of the target vehicle based on a similar calculation method as described above.

[0165] In some embodiments, the processor may determine the relative position information of the target object relative to the target vehicle in a variety of ways based on the depth information of the target object and the depth information of the target vehicle. For example, the processor may convert the pixel coordinates of the target object into the coordinates of the target object in three-dimensional space based on the depth information of the target object; convert the pixel coordinates of the target vehicle into the coordinates of the target vehicle in three-dimensional space based on the depth information of the target vehicle; and determine the relative position information of the target object relative to the target vehicle based on the coordinates of the target object in three-dimensional space and the coordinates of the target vehicle in three-dimensional space.

[0166] In some embodiments of the present specification, obtaining depth information by calculating parallax using binocular vision technology can provide higher accuracy and help with precise positioning at close ranges; it can help help the target vehicle better adapt to various complex real-world application environments.

[0167] In some embodiments, if the target object is an obstacle, the processor may control the target vehicle to bypass the target object based on the relative position information; if the target object is cargo, the processor may control the target vehicle to travel to the target object based on the relative position information.

[0168] In some embodiments, the processor may adjust the target vehicle's driving route to bypass the obstacle based on the relative position information of the obstacle relative to the target vehicle.

[0169] In some embodiments, the processor may adjust the driving route of the target vehicle to move closer to the cargo based on the relative position information of the cargo relative to the target vehicle.

[0170] In some embodiments, the processor may perform driving planning for the target vehicle based on the relative position information of the target object relative to the target vehicle.

[0171] Driving planning includes route control and vehicle control; route control includes lane change, route planning / adjustment, etc.; vehicle control includes vehicle acceleration, vehicle acceleration, vehicle turning, vehicle U-turn, starting vehicle, stopping vehicle, etc.

[0172] Exemplarily, the processor obtains the image to be detected of the surrounding environment through a binocular camera set around the vehicle (such as directly in front), determines the distance between the AGV and the cargo based on the image to be detected, and automatically controls the AGV vehicle to go to the designated cargo area to complete the corresponding handling and unloading operations.

[0173] In some embodiments of the present specification, the target vehicle can be efficiently and accurately controlled through relative position information, which can improve the safety, efficiency and autonomy of the vehicle.

[0174] In some embodiments of the present specification, a binocular camera can be used to accurately calculate the three-dimensional coordinates of a target object in a world coordinate system, which is more accurate than using only the two-dimensional information provided by a monocular camera or a lidar or other sensor, and helps the vehicle to perceive the surrounding environment more accurately. Based on an improved target detection model (such as YOLO, etc.), the target detection task can be completed efficiently and accurately, and combined with the real-time image input of the binocular camera, real-time positioning of the target object can be achieved.

[0175] Figure 3 is an exemplary flowchart for determining a detection object and a target vehicle according to some embodiments of this specification.

[0176] In some embodiments, the target detection model includes a first network, a second network, and a third network connected in sequence. Before detecting objects and identifying target vehicles based on monocular images, it is necessary to first obtain a pre-trained initial model to detect objects and identify target vehicles based on monocular images using the pre-trained initial model. The initial model is a target detection model trained using multiple training samples.

[0177] In some embodiments, the processor may determine the detected object and the target vehicle through process 300 based on the target detection model.

[0178] In some embodiments, process 300 may be implemented by a processor of an object positioning system. Figure 3 As shown, process 300 includes the following steps:

[0179] Step S310, using the optimized convolution module of the first network to extract features of the target object in the monocular image to obtain feature information of multiple scales.

[0180] The first network is used to extract features of different scales from the input image to be detected.

[0181] Feature information may include, but is not limited to, any one or combination of edge features, texture features, color features, shape features, local structures, etc.

[0182] The optimized convolution module can adjust the size and position of the convolution kernel to better adapt to the shape and posture changes of the target object.

[0183] In some embodiments, the backbone network includes multiple layers, the first layer of the backbone network uses a standard convolution operation for initial feature extraction, and in subsequent layers, an optimized convolution module is introduced.

[0184] In some embodiments, feature information of a monocular image can be extracted by adjusting the size and position of the convolution kernel based on at least one deformable convolution module in the optimized convolution module. For more information about the deformable convolution module, see Figure 4 Related description.

[0185] Step S320, using the attention mechanism of the second network to determine the weight of each feature information, and based on the weight, perform feature enhancement on multiple feature information to obtain multiple enhanced features; based on the multiple enhanced features, obtain multiple fusion features.

[0186] The second network is used to fuse features of different scales and output fused features of different scales.

[0187] The attention mechanism dynamically adjusts the weight of each feature information by increasing the weight of important parts (such as the size, shape, and outline of the object) and reducing the weight of unimportant parts (such as irrelevant background information) when processing information.

[0188] Enhanced features refer to feature maps processed by the attention mechanism.

[0189] Fusion features refer to the combination of multiple feature information to form a comprehensive feature representation.

[0190] In some embodiments, the attention mechanism may include a channel attention mechanism, a spatial attention mechanism, etc. The channel weight may be obtained through the channel attention mechanism, and the spatial weight may be obtained through the spatial attention mechanism.

[0191] In some embodiments, the feature map can be adjusted through a channel attention mechanism and a spatial attention mechanism, and the adjusted feature map can be subjected to feature fusion to obtain a fused feature.

[0192] In some embodiments, the attention mechanism is a grouped attention module, which determines the weights of the input feature information in multiple channels based on the grouped attention module, and determines the multiple channel attention features of the feature information by weighting the weights of the multiple channels; and performs feature fusion and residual connection on the multiple channel attention features to obtain enhanced features. For more information about the grouped attention module, please refer to Figure 4 Related description.

[0193] Step S330, using the third network to process multiple fusion features, determine whether there is a detection object and / or a target vehicle in the monocular image, and when there is a detection object and a target vehicle in the monocular image, mark the detection frame of the detection object and / or the target vehicle in the monocular image.

[0194] The third network performs classification regression prediction on fusion features of different scales through different detection heads to obtain detection results.

[0195] In some embodiments, the third network is used to perform target detection and classification tasks on the fused features, and the head network may include a detection head and a classification head. The detection head is used to predict the detection box coordinates of each candidate region (RoI). The classification head is used to predict the category information of each candidate region.

[0196] In some embodiments, the detection head may include a series of convolutional layers and deconvolutional layers to generate a detection frame indicating the location of the object, such as a detection frame of a target object and / or a target vehicle, by predicting the regression value of the detection frame of each anchor frame and the confidence of the existence of the target object.

[0197] In some embodiments, the classification head may classify each feature map using global average pooling, and output a probability distribution of each category, such as a detection box of a target object and / or a target vehicle, by reducing the dimension of the feature map.

[0198] In some embodiments, the target detection model is trained through the following steps: inputting the training sample into the initial model to obtain the predicted detection box of the training sample; determining the loss value of the initial model based on the predicted detection box and the label detection box of the training sample; training the initial model based on the loss value to obtain the target detection model.

[0199] In some embodiments, the training is performed based on a large number of labeled training samples through various feasible methods. For example, parameter updating can be performed based on the gradient descent method. An exemplary training process includes: inputting a plurality of labeled training samples into the initial model, constructing a loss function through the label detection box and the predicted detection box of the initial model, and iteratively updating the parameters of the initial effect prediction model through gradient descent or other methods based on the loss value of the loss function. When the preset conditions are met, the model training is completed and a trained effect prediction model is obtained. Among them, the preset conditions can be that the loss function converges, the number of iterations reaches a threshold, etc.

[0200] In some embodiments, the first training sample includes at least sample images, and the sample images may include two categories: one category of images is sample images labeled with objects of different categories, and the other category is sample images without objects. The training sample may be obtained based on historical data.

[0201] In some embodiments, the label detection frame may include the actual category and the actual detection frame corresponding to the sample image. The label detection frame may be obtained by a processor or manual annotation.

[0202] In some embodiments, the loss function may be any one or a combination of a mean square error loss function, a cross entropy loss function, etc.

[0203] In some embodiments of the present specification, a loss function is constructed by predicting a detection box and a label detection box, which helps the target detection model learn the relationship between each feature in the input image and the detected object, thereby improving the accuracy of the model output.

[0204] In some embodiments, an initial intersection-and-union ratio can be determined based on the predicted detection box and the label detection box; based on the size relationship between the initial intersection-and-union ratio and a preset threshold, the initial intersection-and-union ratio is normalized to obtain a target intersection-and-union ratio; and a loss value is determined based on the target intersection-and-union ratio.

[0205] The initial intersection-over-union ratio measures the degree of overlap between the predicted detection box and the label detection box.

[0206] The target intersection-and-union ratio refers to the intersection-and-union ratio after linear mapping.

[0207] In some embodiments, based on the magnitude relationship between the initial IoU and a preset threshold, the initial IoU is normalized in a variety of ways to obtain a target IoU. For example, the initial IoU may be a linear function of the target IoU.

[0208] In some embodiments of the present specification, by giving greater weight to samples that are difficult to classify based on the size relationship between the initial intersection-over-union ratio and a preset threshold, it helps to solve the problem of class imbalance in target detection, makes the model pay more attention to targets that are difficult to classify, and improves the detection ability of small targets and rare categories.

[0209] In some embodiments, if the initial intersection-and-union ratio is less than a first threshold, the target intersection-and-union ratio is set to a first preset value; if the initial intersection-and-union ratio is less than a second threshold, the target intersection-and-union ratio is set to a second preset value; if the initial intersection-and-union ratio is within the range of the first threshold and the second threshold, the initial intersection-and-union ratio is normalized to obtain the target intersection-and-union ratio.

[0210] In some embodiments, the loss function is a Focaler-IoU loss function.

[0211] In some embodiments, a linear interval mapping method can be used to reconstruct the IoU (Intersection over Union, IoU) loss and improve edge regression. The reconstructed focer-IoU calculation is shown in formula (2):

[0212]

[0213] Among them, IoU focaler is the reconstructed focer-IoU, i.e., the target intersection-union ratio, and IoU is the original IoU value, [d,u]∈[0,1]. By adjusting the values ​​of d and u, IoU focaler The focus is on different regression samples, and the loss is defined as formula (3):

[0214] L Focaler-IoU =1-IoU focaler ; (3)

[0215] Among them, L Focaler-IoU is the loss function, based on L Focaler-IoU The value calculated by the loss function is the loss value.

[0216] In some embodiments of the present specification, the Focaler-IoU loss function is combined with the IoU Loss to measure the overlap between the generated bounding box and the true bounding box. By optimizing the IoU Loss, the Focal-IoU loss function can prompt the model to generate a more accurate bounding box and improve the quality and accuracy of the bounding box.

[0217] In some embodiments of the present specification, by optimizing the convolution module and the attention mechanism, the target detection model can better capture the key features of the target object, improve the accuracy and robustness of the detection, enable the target detection model to better adapt to target objects of different shapes, scales and postures, improve the generalization ability of the model, and improve the performance and efficiency of the target detection task.

[0218] It should be noted that the above description of the relevant process is only for example and explanation, and does not limit the scope of application of this specification. For those skilled in the art, various modifications and changes can be made to the process under the guidance of this specification. However, these modifications and changes are still within the scope of this specification.

[0219] Figure 4 is an exemplary schematic diagram of a target detection model according to some embodiments of the present specification.

[0220] In some embodiments, the target detection model can be obtained based on the improvement of the YOLOv8 model. The monocular image is processed by the trained target detection model to obtain the detected object and / or target vehicle, which specifically includes the following steps:

[0221] Step S410: Input the monocular image into the initial model, perform multi-scale convolution processing through the first network, and obtain the first feature, the second feature and the third feature of the monocular image.

[0222] like Figure 4 As shown, the first network includes a first feature layer, a second feature layer and a third feature layer connected in sequence, and the first feature layer, the second feature layer and the third feature layer all include a convolution module and an optimized convolution module, wherein the third feature layer adds a spatial pyramid pooling fast (SPPF) module. Figure 4 As shown, the first feature layer includes two convolution modules, an optimized convolution module, a convolution module, and an optimized convolution module connected in sequence. The second feature layer includes a convolution module and an optimized convolution module connected in sequence, and the third feature layer includes a convolution module, an optimized convolution module, and an SPPF module connected in sequence.

[0223] In some embodiments, the first network performs feature extraction on a monocular image, including: inputting the monocular image into an initial model, performing feature processing on the monocular image through a first feature layer of the first network to obtain a first feature output by the first feature layer; performing feature processing on the first feature output by the first feature layer through a second feature layer to obtain a second feature output by the second feature layer; performing feature processing on the second feature output by the second feature layer through a third feature layer to obtain a third feature output by the third feature layer.

[0224] In some embodiments, the feature processing at the first feature layer includes: convolution of two convolution modules, feature optimization of the optimized convolution module, re-convolution of the convolution module, and re-optimization of the optimized convolution module, and outputting the first feature. The feature processing at the second feature layer includes re-convolution of the convolution module and re-optimization of the features of the optimized convolution module, and outputting the second feature. The feature processing at the third feature layer includes re-convolution of the convolution module, re-optimization of the features of the optimized convolution module, and fast spatial pyramid pooling of the SPPF module, and outputting the third feature.

[0225] For more information on optimizing convolution modules, see Figure 5 Related description.

[0226] The convolution module is used to perform convolution operations on the input image to extract local features of the image.

[0227] In some embodiments, the convolution module may include at least one or any combination of a convolution kernel, a normalization function, an activation function, etc. For example, the convolution module may be a CBS (Convolution-Batch Normalization-SiLU) module, wherein the size of the convolution kernel may be 3×3, the normalization function may be a BN (Batch Normalization) function, and the activation function may be a relu (Rectified Linear Unit) function, a Sigmoid function, etc.

[0228] The spatial pyramid pooling module can extract features of different scales by performing pooling operations on feature maps at different scales.

[0229] In some embodiments, multiple feature layers of the first network include convolution modules and optimized convolution modules, wherein the third feature layer also adds a spatial pyramid pooling module SPPF, which can perform multi-scale feature extraction through the first network, improve feature diversity and improve the real-time performance of feature extraction, and provide a solid foundation for subsequent multi-scale-based feature prediction.

[0230] Figure 5 is an exemplary schematic diagram of an optimized convolution module according to some embodiments of the present specification.

[0231] like Figure 5 As shown, in some embodiments, the optimized convolution module includes two deformable convolution modules, multiple Bottleneck layers, and a splicing module.

[0232] In some embodiments, based on optimizing at least one deformable convolution module in the convolution module, extracting feature information of the monocular image by adjusting the size and position of the convolution kernel includes:

[0233] The input of the first deformable convolution module is the input of the optimized convolution module;

[0234] The output end of the first deformable convolution module is connected to the input end of the bottleneck layer; the bottleneck layer extracts features through multiple serially connected Bottleneck layers; the output end of the first deformable convolution module, the output end of the bottleneck layer, and the output end of each Bottleneck layer are respectively connected to the input end of the splicing module, and spliced ​​through the splicing module to obtain a spliced ​​feature map; the spliced ​​feature map is processed through the second deformable convolution module to obtain feature information of the output of the optimized convolution module.

[0235] In some embodiments, the deformable convolution module may adopt DCN (Deformable Convolution) v4.

[0236] The internal structure of each Bottleneck layer can be exactly the same or different. The Bottleneck layer can be used to perform 1*1 convolution operations.

[0237] The input end of the first deformable convolution module is the input end of the optimized convolution module, and the deformable convolution module performs initialization feature extraction by dynamically adjusting the shape of the convolution kernel.

[0238] The output of the first deformable convolution module includes a first feature part and a second feature part. The first feature part is used to pass to the bottleneck layer, and the second feature part is used as the input of the subsequent splicing module.

[0239] The bottleneck layer includes one or more serially connected Bottleneck layers to extract deeper features. The Bottleneck layer can reduce the number of parameters and the amount of calculation of the input feature map.

[0240] The concatenation module (Concatenation Layer) is used to concatenate the first feature part, the output of the bottleneck layer, and the output of all intermediate bottleneck layers to obtain a concatenated feature map.

[0241] By splicing the output of the first deformable convolution module of the splicing module with the feature maps processed by each Bottleneck layer and the output of the bottleneck layer, multi-level feature information can be integrated.

[0242] The second deformable convolution module is used to further fuse and process the spliced ​​feature maps output by the splicing module.

[0243] It should be noted that “first”, “second”, etc. are only used for the convenience of distinction. For example, the structures of the first deformable convolution module and the second deformable convolution module can be exactly the same, and the same applies to others.

[0244] In some embodiments of the present specification, the efficiency and effectiveness of computer vision applications can be improved through DCNv4; DCNv4 helps to extract rich image texture features in various visual tasks, such as image classification and generation, and has faster convergence speed and higher processing efficiency.

[0245] Step S420: The first feature, the second feature, and the third feature are fused through the second network to obtain a first fused feature, a second fused feature, and a third fused feature.

[0246] In some embodiments, the second network of the traditional YOLOvn8 model includes an upsampling structure and a downsampling structure, and a grouping attention module can be added between the upsampling structure and the downsampling structure in the second network of the traditional YOLOvn8 model. For example, at least two of the first feature, the second feature, and the third feature can be upsampled and concatenated to obtain three initial features, and the three initial features are input into the first fusion layer, the second fusion layer, and the third fusion layer respectively. The first fusion layer, the second fusion layer, and the third fusion layer all include a grouping attention module to perform feature fusion processing respectively to obtain a first fusion feature output by the first fusion layer, a second fusion feature output by the second fusion layer, and a third fusion feature output by the third fusion layer.

[0247] Step S430: predict the first fusion feature, the second fusion feature and the third fusion feature respectively through the third network to obtain the detection object and / or the target vehicle.

[0248] After obtaining the first fusion feature, the second fusion feature and the third fusion feature, the first fusion feature, the second fusion feature and the third fusion feature are respectively identified and predicted through the third network to obtain the identification data output by the third network, that is, the detected object and / or target vehicle.

[0249] In some embodiments, the third network includes a first detection module, a second detection module, and a third detection module; wherein,

[0250] The first detection module detects the first fused feature of the second network and outputs target 1; the second detection module detects the second fused feature of the second network and outputs target 2; the third detection module detects the third fused feature of the second network and outputs target 3; finally, target 1, target 2, and target 3 are fused to output the final detection frame coordinates and category information of the detected object and / or target vehicle.

[0251] In this embodiment, through the second network, multi-scale fusion features can be more accurately fused, thereby more accurately identifying the detected object and / or target vehicle, providing an effective data basis for subsequent detection of the object and / or target vehicle coordinate positioning.

[0252] In one embodiment, the second network includes an initial fusion layer, and a first fusion layer, a second fusion layer, and a third fusion layer connected in sequence. Specifically, in step S420, the first feature, the second feature, and the third feature are fused by the second network, including:

[0253] S421: Fusing at least two of the first feature, the second feature, the third feature, etc. through an initial fusion layer to obtain a first initial feature, a second initial feature, and a third initial feature.

[0254] The initial fusion layer includes multiple upsampling modules, multiple splicing modules and optimized convolution modules. The fusion processing of the initial fusion layer includes: taking the third feature output by the spatial pyramid pooling module as the third initial feature, passing the third feature through an upsampling layer, and then performing a splicing operation with the second feature in the first splicing module; the feature map output by the first splicing module is passed through an optimized convolution module to extract features and output the second initial feature. After the second initial feature passes through an upsampling layer, it is spliced ​​with the first feature in the second splicing module to obtain the first initial feature output by the second splicing module.

[0255] The initial fusion layer can improve the accuracy of the features output by each subsequent feature fusion layer.

[0256] S422: The first initial feature is fused by the first fusion layer, the second initial feature is fused by the second fusion layer, and the third initial feature is fused by the third fusion layer to obtain the first fused feature, the second fused feature and the third fused feature.

[0257] The first fusion layer passes the first initial feature through the grouping attention module and the optimized convolution module of the first fusion layer, and then extracts the feature to output the first fusion feature.

[0258] The second fusion layer passes the first fusion feature through the convolution module of the second fusion layer, and then performs a splicing operation with the second initial feature in the splicing module. The feature map output by the splicing module passes through a grouping attention module and an optimized convolution module in turn to extract features and output the second fusion feature;

[0259] The third fusion layer passes the second fused feature through the convolution module of the third fusion layer, and then performs a splicing operation with the third initial feature output by the spatial pyramid module in the splicing module; the feature map output by the splicing module is extracted through a grouping attention module and an optimized convolution module in turn, and the third fused feature is output.

[0260] Figure 6 It is an exemplary schematic diagram of the grouping attention module shown in some embodiments of the present specification.

[0261] like Figure 6 As shown, in some embodiments, the group attention module includes multiple convolutional layers, a channel attention module, multiple weight layers, and a residual splicing module.

[0262] In some embodiments, the weights of input feature information in multiple channels are determined based on a grouped attention module, including: performing a segmentation operation on a feature map input to the grouped attention module to obtain a feature map of a first branch and a feature map of a second branch; performing different processing on the feature map of the first branch and the feature map of the second branch respectively to obtain a processed feature map of the first branch and a processed feature map of the second branch; based on the processed feature map of the first branch and the processed feature map of the second branch, extracting weights through a channel attention module, and outputting channel weights of the first branch and channel weights of the second branch.

[0263] Among them, the feature map of the input group attention module is segmented to obtain the feature map of the first branch and the feature map of the second branch, including: inputting the feature map of the input group attention module into a convolutional layer (e.g., 1×1Conv), and performing a segmentation operation on the result of the convolutional layer to obtain a first feature map of the first branch and a second feature map of the second branch.

[0264] The step of performing different processing on the feature map of the first branch and the feature map of the second branch to obtain the processed feature map of the first branch and the processed feature map of the second branch includes:

[0265] Input the first feature map into a convolutional layer (e.g., 3×3Conv) to obtain a third feature map;

[0266] Input the second feature map into a convolutional layer (e.g., 3×3Conv) to obtain a fourth feature map;

[0267] Superimposing the third characteristic map and the fourth characteristic map to obtain a fifth characteristic map;

[0268] The fifth feature map is input into a convolutional layer (e.g., 3×3Conv) to obtain a sixth feature map.

[0269] The third feature map and the sixth feature map are the feature map of the processed first branch and the feature map of the processed second branch, respectively.

[0270] Wherein, based on the processed feature map of the first branch and the processed feature map of the second branch, the weights are extracted through the channel attention module, and the channel weights of the first branch and the channel weights of the second branch are output, including:

[0271] Based on the third feature map and the sixth feature map, the channel weight of the first branch and the channel weight of the second branch are obtained through the channel attention module. For information on obtaining the channel weight of the first branch and the channel weight of the second branch, please refer to Figure 6 The relevant description is as follows.

[0272] By adopting two branches to determine the channel weights of different branches, the number of parameters can be reduced.

[0273] In some embodiments, the weights of multiple channels are weighted to determine multiple channel attention features of feature information, including:

[0274] The third feature map and the channel weight of the first branch are weighted to obtain the seventh feature map, and the sixth feature map and the channel weight of the second branch are weighted to obtain the eighth feature map;

[0275] The seventh feature map and the eighth feature map are used as multiple channel attention features of the two branches.

[0276] In some embodiments, feature fusion and residual connection are performed on multiple channel attention features to obtain enhanced features, including:

[0277] Input the seventh feature map and the eighth feature map into a convolutional layer (e.g., 1×1Conv) to obtain a ninth feature map;

[0278] The ninth feature map and the original feature map are input into the residual concatenation module to obtain the output of the group attention module as the enhanced feature.

[0279] Among them, the convolution layer can perform convolution operations to capture the features of the input image. It should be noted that the convolution layer is the basic building block of the convolution module, and the convolution module is a modular structure that can be composed of multiple convolution layers and other components.

[0280] The segmentation operation is used to segment the output of the eighth convolutional layer to obtain two feature maps after segmentation, namely the first feature map of the first branch and the second feature map of the second branch. The two branches will be processed through different paths respectively.

[0281] The superposition operation is used to superimpose the third feature map and the fourth feature map to obtain a fifth feature map.

[0282] The channel attention module is used to calculate the third feature map and the sixth feature map to obtain the channel weight of the first branch and the channel weight of the second branch.

[0283] The first weight layer uses the channel weight of the first branch to perform weighted calculation on the third feature map to obtain the seventh feature map; the second weight layer uses the channel weight of the second branch to perform weighted calculation on the sixth feature map to obtain the eighth feature map.

[0284] The residual splicing module is used to superimpose the ninth feature map with the original input feature map to obtain the final output feature map.

[0285] In some embodiments of the present specification, in the second network, adding a group attention module can obtain more subtle information and solve the problem of premature loss of image detail information during the downsampling process, thereby improving the target detection performance and maintaining high-precision and real-time target detection when the parameter amount is extremely small.

[0286] In some embodiments, the channel attention module includes a pooling layer, multiple convolutional layers, and an activation layer, and the channel attention module is used to calculate the importance of different feature maps.

[0287] In some embodiments, the processed feature map of the first branch and the processed feature map of the second branch can be superimposed to obtain a fused feature map; the fused feature map can be pooled to obtain global information of multiple channels; based on the global information of multiple channels, the channel weight of the first branch and the channel weight of the second branch can be determined.

[0288] Among them, the processed feature map of the first branch and the processed feature map of the second branch are superimposed to obtain a fused feature map, including: the channel attention module can superimpose the third feature map and the sixth feature map (such as splicing based on the channel dimension, etc.) to obtain a fused feature map.

[0289] The fused feature map is pooled to obtain global information of multiple channels, including: the fused feature map is pooled by global average of the pooling layer to obtain global information of multiple channels.

[0290] For example, the fused feature map can be input into a pooling formula to obtain global information of multiple channels. The pooling formula is formula (5):

[0291]

[0292] Among them, S c is the global information of the cth channel, is the fused feature map, H and W are the height and width of the fused feature map, respectively, and C is the number of channels.

[0293] By extracting global information and avoiding the loss of feature information, it helps to obtain the key information of the feature map.

[0294] In some embodiments, based on the global information of multiple channels, the channel weight of the first branch and the channel weight of the second branch are determined, including: passing the global information of multiple channels through convolution operations of two convolutional layers in sequence to obtain local information of the first branch and local information of the second branch; normalizing the local information of the first branch and the local information of the second branch to obtain the channel weight of the first branch and the channel weight of the second branch.

[0295] The global information of multiple channels is sequentially passed through the convolution operation of two convolutional layers to obtain the local information of the first branch and the local information of the second branch, including: the global information of multiple channels is sequentially passed through the convolution operation of the first convolutional layer and the convolution operation of the second convolutional layer to obtain the local information of the first branch and the local information of the second branch.

[0296] Among them, the local information of the first branch and the local information of the second branch are normalized to obtain the channel weight of the first branch and the channel weight of the second branch, including: the local information of the first branch and the local information of the second branch are activated by an activation layer to obtain the channel weight of the first branch and the channel weight of the second branch.

[0297] Exemplarily, the activation process may be implemented based on a normalization process formula, and the normalization process formula is formula (7):

[0298]

[0299] in, is the local information of the i-th branch in the c-th channel, i = 1, 2; a i (c) is the channel weight of the i-th branch in the c-th channel.

[0300] Through the channel weights of the two branches, splitting the weighted combination can help capture the key information in the feature map.

[0301] In some embodiments, the first convolution layer may include a 1x1 Conv and a normalization layer (e.g., BatchNorm) and an activation function layer (e.g., ReLU function) connected in sequence, and the second convolution layer may be a 1x1 Conv.

[0302] In some embodiments, the pooling layer is a global pooling layer (eg, Global Pooling).

[0303] In some embodiments, the activation layer may employ a SoftMax function.

[0304] In some embodiments of the present specification, the connection and path cascade of the feature pyramid network are enhanced through the attention mechanism, and a more powerful multi-scale feature representation is provided, so that the network can better capture complex semantic information. The feature pyramid can handle the multi-scale variation problem in target recognition without increasing the computational overhead; the extracted features can generate multi-scale feature representations, including high-resolution features.

[0305] Figure 7 This is an exemplary schematic diagram of another grouping attention module shown in some embodiments of the present specification.

[0306] like Figure 7 As shown, in some embodiments, the group attention module uses two branches (N=2) to reduce the number of parameters. The two branches include Split1 and Split2. Both branches include a 1x1 Conv and a 3x3 Conv. In Split2, the fourth feature map output by the tenth convolutional layer is summed with the third feature map output by the ninth convolutional layer in Split1, and the sum is input into the eleventh convolutional layer (e.g., 3x3 Conv). By receiving semantic information from the two split groups, the network's receptive field is expanded. The sum of the third feature map and the fourth feature map (the middle sum) can be determined by formula (4):

[0307]

[0308] in, is the sum of the third and fourth feature maps, X i Represents the input image information, F i Represented as the result information of the output feature map of Split(i), i=1,2.

[0309] The channel statistics generated by the first pooling layer collect the global spatial information as formula (5):

[0310]

[0311] α and β represent the point-by-point spatial information of the feature map in the height and width directions, respectively.

[0312] The channel attention mechanism is used to aggregate the weighted fusion represented by the basis array, where the split weighted combination can capture the key information in the feature map. The feature map of the cth channel is calculated as formula (6):

[0313]

[0314] a i Assign a (soft) weight to i It can be determined by formula (7):

[0315]

[0316] in, Represents the weight of the global spatial information S on the cth channel, which is quantized by two 1x1Conv (BatchNorm and ReLU activation).

[0317] In some embodiments, a standard residual architecture (ResNet) is used, and the ninth feature map is summed with the original feature map input into the residual concatenation module to obtain the output of the grouped attention module. Figure 7 In , + indicates the concatenation of channel dimensions, such as c' / 2+c' / 2=c'. X indicates weighted calculation of input dimensional information or spatial information to extract important information.

[0318] In some embodiments of the present specification, in the second network, upsampling and downsampling are completed by linear interpolation, and multi-scale feature maps are directly added to complete fusion, which can reduce the amount of calculation, but feature information will be lost. A grouping attention module is added between the upsampling result and the downsampling result to screen the features lost in the linear interpolation process, extract the area that the network pays most attention to, and reduce model loss; the grouping attention module helps to enhance the connection and path cascading of the feature pyramid network.

[0319] It can be understood that the network structure of the target detection model, or the modules included in the network structure, do not constitute a limitation on the embodiments of the present application. That is, the number of models or types of models included in the target detection model, or the number and type of convolution modules or other modules included in each model do not affect the overall implementation of the technical solution in the embodiments of the present application, and can all be regarded as equivalent replacements or derivatives of the technical solution claimed to be protected in the embodiments of the present application.

[0320] Figure 8 is an exemplary module diagram of an object positioning system according to some embodiments of the present specification.

[0321] like Figure 8 As shown, one or more embodiments of this specification further provide an object positioning system 800, the system comprising:

[0322] A first detection module 810 is used to process the collected monocular image by using a trained target detection model to obtain a detection object and a target vehicle in the monocular image;

[0323] A second detection module 820, configured to obtain a first binocular image of the target object and a second binocular image of the target vehicle if the detected objects include the target object;

[0324] The positioning module 830 is used to determine the relative position information of the target object with respect to the target vehicle based on the first binocular image and the second binocular image.

[0325] The specific implementation of each of the above modules can be found in the previous embodiments and will not be described in detail here.

[0326] Fig. 9 is a schematic diagram of the structure of an electronic device according to some embodiments of this specification. Fig. 9 As shown, the electronic device 900 may include: a processor 901, a memory 902. The electronic device 900 may also include one or more of a multimedia component 903, an input / output (I / O) component 904, and a communication component 905. In this embodiment, the electronic device 900 may be a device integrated in a vehicle to interact with the aforementioned object positioning system to implement the object positioning method provided in this embodiment. It should be understood that the electronic device 900 may also include some components of the aforementioned object positioning system.

[0327] The processor 901 is used to control the overall operation of the electronic device 900 to complete all or part of the steps in the above-mentioned object positioning method. The memory 902 is used to store various types of data to support the operation of the auxiliary electronic device 900. These data may include instructions for any application or method used to operate on the auxiliary electronic device 900, as well as application-related data, such as contact data, sent and received messages, pictures, audio, video, etc. The memory 902 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random Access Memory, SRAM), electrically erasable programmable read only memory (Electrically Erasable Programmable Read Only Memory, EEPROM), erasable programmable read only memory (Erasable Programmable Read Only Memory, EPROM), programmable read only memory (Programmable Read Only Memory, PROM), read only memory (Read Only Memory, ROM), magnetic memory, flash memory, magnetic disk or optical disk. The multimedia component 903 may include a screen and an audio component. The screen may be, for example, a touch screen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signal may be further stored in the memory 902 or sent through the communication component 905. The audio component also includes a speaker for outputting audio signals. The I / O component 904 provides an interface between the processor 901 and other interface modules, and the above-mentioned other interface modules may be keyboards, mice, buttons, etc. These buttons may be virtual buttons or physical buttons. The communication component 905 is used for wired or wireless communication between the electronic device 900 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, 4G, Narrow Band Internet of Things (NB-IOT), Enhanced Narrow Band Internet of Things technology (Enhanced Machine-Type Communication, eMTC), or other 5G, etc., or a combination of one or more of them, is not limited here. Therefore, the corresponding communication component 905 may include: Wi-Fi module, Bluetooth module, NFC module, etc.

[0328] In an exemplary embodiment, the electronic device 900 can be implemented by one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), controllers, microcontrollers, microprocessors or other electronic components to execute the above-mentioned object positioning method.

[0329] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, and when the program instructions are executed by a processor, the steps of the above-mentioned object positioning method are implemented. For example, the computer-readable storage medium may be the above-mentioned memory 902 including program instructions, and the above-mentioned program instructions may be executed by the processor 901 of the electronic device 900 to complete the steps of the object positioning method provided in any of the above-mentioned embodiments;

[0330] Alternatively, when the instructions are executed by a computer, the computer executes the steps of the object positioning method provided in any of the above embodiments.

[0331] The present application also provides a vehicle, on which is disposed the object positioning system provided by any one of the above embodiments, and the object positioning system is used to execute the object positioning method provided by any one of the above embodiments.

[0332] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0333] The embodiments, implementation methods and related technical features of the present application can be combined and replaced with each other without conflict.

[0334] The above are only preferred embodiments of the present application and do not constitute any form of limitation to the present application. Although the descriptions of various embodiments in the embodiments of the present application have different focuses, for parts not described in detail in a certain embodiment, reference can be made to the relevant embodiments of other embodiments. However, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present application without departing from the content of the technical solution of the present application are still within the scope of the technical solution of the present application.

Claims

1. A method for locating an object, characterized in that: The method comprises: The collected monocular image is processed by a trained target detection model to obtain the detection object and the target vehicle in the monocular image; If the detected objects include a target object, acquiring a first binocular image of the target object and a second binocular image of the target vehicle; Based on the first binocular image and the second binocular image, relative position information of the target object relative to the target vehicle is determined.

2. The method according to claim 1, characterized in that: The target detection model is trained by the following steps: inputting a training sample into an initial model to obtain a predicted detection frame of the training sample; Determining a loss value of the initial model according to the predicted detection frame and the label detection frame of the training sample; The initial model is trained according to the loss value to obtain the target detection model.

3. The method according to claim 2, characterized in that The determining the loss value of the initial model according to the predicted detection frame and the label detection frame of the training sample includes: Determining an initial intersection-over-union ratio based on the predicted detection frame and the label detection frame; Based on the magnitude relationship between the initial intersection-and-union ratio and a preset threshold, the initial intersection-and-union ratio is normalized to obtain a target intersection-and-union ratio; Based on the target intersection-over-union ratio, the loss value is determined.

4. The method according to claim 3, characterized in that The step of normalizing the initial intersection-and-union ratio based on the size relationship between the initial intersection-and-union ratio and a preset threshold value to obtain a target intersection-and-union ratio includes: If the initial intersection-over-union ratio is less than a first threshold, setting the target intersection-over-union ratio to a first preset value; If the initial intersection-over-union ratio is less than a second threshold, setting the target intersection-over-union ratio to a second preset value; If the initial intersection-over-union ratio is within the range of the first threshold and the second threshold, the initial intersection-over-union ratio is normalized to obtain the target intersection-over-union ratio.

5. The method according to claim 1, characterized in that The target detection model includes a first network, a second network and a third network connected in sequence; The method of processing the collected monocular image by the trained target detection model to obtain the detection object and the target vehicle in the monocular image includes: Using the optimized convolution module of the first network to extract features of the target object in the monocular image, and obtain feature information of multiple scales; Determine the weight of each of the feature information by using the attention mechanism of the second network, and enhance the features of the plurality of feature information based on the weights to obtain a plurality of enhanced features; and obtain a plurality of fused features based on the plurality of enhanced features; The third network is used to process the multiple fusion features to determine whether the detection object and the target vehicle exist in the monocular image, and when the detection object and / or the target vehicle exist in the monocular image, a detection frame of the detection object and / or the target vehicle is marked in the monocular image.

6. The method according to claim 5, characterized in that The optimized convolution module of the first network is used to extract features of the target object in the monocular image to obtain feature information of multiple scales, including: Based on at least one deformable convolution module in the optimized convolution module, feature information of the monocular image is extracted by adjusting the size and position of the convolution kernel.

7. The method according to claim 5, characterized in that The attention mechanism is a grouped attention module, which uses the attention mechanism of the second network to determine the weight of each of the feature information, and enhances the multiple feature information based on the weight to obtain multiple enhanced features, including: Based on the grouped attention module, the weights of the input feature information in multiple channels are determined, and the weights of the multiple channels are weighted to determine multiple channel attention features of the feature information; feature fusion and residual connection are performed on the multiple channel attention features to obtain the enhanced features.

8. The method according to claim 7, characterized in that The step of determining the weights of the input feature information in multiple channels based on the grouped attention module includes: Perform a segmentation operation on the feature map of the input group attention module to obtain a feature map of the first branch and a feature map of the second branch; Performing different processing on the feature graph of the first branch and the feature graph of the second branch respectively to obtain a processed feature graph of the first branch and a processed feature graph of the second branch; Based on the processed feature map of the first branch and the processed feature map of the second branch, weights are extracted through a channel attention module, and the channel weights of the first branch and the channel weights of the second branch are output.

9. The method according to claim 8, characterized in that The step of extracting weights through a channel attention module based on the processed feature map of the first branch and the processed feature map of the second branch, and outputting the channel weights of the first branch and the channel weights of the second branch includes: Superimposing the processed feature map of the first branch and the processed feature map of the second branch to obtain a fused feature map; Performing a pooling operation on the fused feature map to obtain global information of multiple channels; Based on the global information of the multiple channels, a channel weight of the first branch and a channel weight of the second branch are determined.

10. The method according to claim 9, characterized in that The pooling operation is performed on the fused feature map to obtain global information of multiple channels, including: The fused feature map is input into the pooling formula to obtain the global information of the multiple channels. The pooling formula is: Among them, S c is the global information of the cth channel, is the fused feature map, H and W are the height and width of the fused feature map respectively, and C is the number of channels.

11. The method according to claim 9, characterized in that The determining, based on the global information of the plurality of channels, the channel weight of the first branch and the channel weight of the second branch comprises: The global information of the multiple channels is sequentially subjected to convolution operations of two convolutional layers to obtain local information of the first branch and local information of the second branch; The local information of the first branch and the local information of the second branch are normalized to obtain the channel weight of the first branch and the channel weight of the second branch. The formula for the normalization is: in, is the local information of the i-th branch in the c-th channel, i = 1, 2; a i (c) is the channel weight of the i-th branch in the c-th channel.

12. The method according to claim 1, characterized in that The determining, based on the first binocular image and the second binocular image, the relative position information of the target object relative to the target vehicle comprises: Inputting the first binocular image and the second binocular image into the target detection model respectively, obtaining the pixel coordinates of the target object in the first binocular image and the pixel coordinates of the target vehicle in the second binocular image; Based on the pixel coordinates in the first binocular image and the pixel coordinates in the second binocular image, relative orientation information of the target object relative to the target vehicle is determined.

13. The method according to claim 12, characterized in that The relative position information of the target object relative to the target vehicle is determined based on the pixel coordinates in the first binocular image and the pixel coordinates in the second binocular image: Determining depth information of the target object based on pixel coordinates of the target object in the first binocular image and camera calibration parameters; Determining depth information of the target vehicle based on pixel coordinates of the target vehicle in the second binocular image and the camera calibration parameters; Based on the depth information of the target object and the depth information of the target vehicle, relative position information of the target object with respect to the target vehicle is determined.

14. The method according to claim 13, characterized in that The determining the depth information of the target object based on the pixel coordinates of the target object in the first binocular image and the camera calibration parameters includes: Determining a disparity of the target object in the first binocular image based on pixel coordinates of the target object in the first binocular image; The product of the focal length and the center distance of the camera calibration parameters is calculated, and the depth information of the target object is determined based on the ratio of the product to the parallax.

15. The method according to claim 1, characterized in that The method further comprises: Acquire multiple chessboard images of preset chessboard grids at different angles and orientations; Determine the pixel coordinates of each corner point in the chessboard image; Determine the projection coordinates of the corner point in the world coordinate system; A camera calibration parameter is determined based on the projection coordinates and the pixel coordinates.

16. The method according to any one of claims 1 to 15, characterized in that: After determining the relative position information of the target object relative to the target vehicle based on the first binocular image and the second binocular image, the method further includes: If the target object is an obstacle, controlling the target vehicle to bypass the target object according to the relative position information; If the target object is cargo, the target vehicle is controlled to travel to the target object according to the relative position information.

17. An object positioning system, characterized in that: The system comprises: A first detection module is used to process the collected monocular image through a trained target detection model to obtain a detection object and a target vehicle in the monocular image; A second detection module, configured to obtain a first binocular image of the target object and a second binocular image of the target vehicle if the detected objects include the target object; A positioning module is used to determine the relative position information of the target object relative to the target vehicle based on the first binocular image and the second binocular image.

18. A computer-readable storage medium having a computer program stored thereon, characterized in that: The computer-readable storage medium stores instructions, which, when executed by a computer, enable the computer to implement the object positioning method according to any one of claims 1 to 16.

19. An electronic device, characterized in that: include: a memory having a computer program stored thereon; A processor, configured to execute the computer program in the memory to implement the object positioning method according to any one of claims 1 to 16.

20. A vehicle, characterized in that: An object positioning system comprising the object positioning system of claim 17.

Images