Target detection method, device and equipment, computer readable storage medium and computer program product

Abstract

The invention provides a target detection method, device and equipment, a computer readable storage medium and a computer program product. The method comprises the steps that three-dimensional data to be detected and a trained three-dimensional detection model are acquired, and the trained three-dimensional detection model is obtained by training the three-dimensional detection model to be trained through three-dimensional sample data; the first model parameter of the to-be-trained three-dimensional detection model is obtained by fusing the second model parameter of the trained two-dimensional detection model; wherein the first model parameter is a model parameter of a to-be-trained three-dimensional detection model, the trained two-dimensional detection model can perform target detection on a two-dimensional image, and the second model parameter is a model parameter of the trained two-dimensional detection model; performing feature extraction on to-be-detected three-dimensional data by using the trained three-dimensional detection model to obtain feature information; and performing prediction processing on the to-be-detected three-dimensional data according to the feature information to obtain a target detection result. According to the invention, the target detection efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision, and in particular to a target detection method, apparatus, device, computer-readable storage medium, and computer program product. Background Art

[0002] Object detection refers to the process of using computer vision technology and algorithms to identify and locate various objects and obstacles from the collected data about the vehicle's surroundings. This data usually comes from various sensors on the vehicle, such as cameras, radars, and LiDAR (Light Detection and Ranging). The purpose of object detection is to enable the autonomous driving system to "see" the surrounding environment so that it can make safe driving decisions.

[0003] In the related art, either only 3D network is used for target detection, or 3D network and 2D network are combined for target detection. When only 3D network is used for target detection, there is a problem of low detection accuracy. When 3D network and 2D network are combined for target detection, the related art needs to align 3D point cloud information and 2D image before model inference can be performed, and both 3D network and 2D network are used during inference, resulting in long inference time, which brings about the problem of low timeliness. Summary of the invention

[0004] The embodiments of the present application provide a target detection method, apparatus, device, computer-readable storage medium, and computer program product, which can improve the efficiency of target detection.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present invention provides a method for detecting a target, which includes:

[0007] Acquire three-dimensional data to be detected and a trained three-dimensional detection model, wherein the trained three-dimensional detection model is obtained by training the three-dimensional detection model to be trained with three-dimensional sample data, and first model parameters of the three-dimensional detection model to be trained are obtained by fusing second model parameters of a trained two-dimensional detection model; wherein the first model parameters are model parameters of the three-dimensional detection model to be trained, the trained two-dimensional detection model can perform target detection on a two-dimensional image, and the second model parameters are model parameters of the trained two-dimensional detection model;

[0008] Using the trained three-dimensional detection model to extract features from the three-dimensional data to be detected to obtain feature information;

[0009] The three-dimensional data to be detected is predicted and processed according to the feature information to obtain a target detection result.

[0010] The present application provides a target detection device, the device comprising:

[0011] A first acquisition module is used to acquire the three-dimensional data to be detected and the trained three-dimensional detection model, wherein the trained three-dimensional detection model is obtained by training the three-dimensional detection model to be trained with the three-dimensional sample data, and the first model parameters of the three-dimensional detection model to be trained are obtained by fusing the second model parameters of the trained two-dimensional detection model; wherein the first model parameters are the model parameters of the three-dimensional detection model to be trained, the trained two-dimensional detection model can perform target detection on the two-dimensional image, and the second model parameters are the model parameters of the trained two-dimensional detection model;

[0012] A first feature extraction module, used to extract features from the three-dimensional data to be detected using the trained three-dimensional detection model to obtain feature information;

[0013] The first prediction module is used to perform prediction processing on the three-dimensional data to be detected according to the feature information to obtain a target detection result.

[0014] An embodiment of the present application provides a computer device, including:

[0015] A memory for storing computer executable instructions;

[0016] The processor is used to implement the target detection method provided in the embodiment of the present application when executing the computer executable instructions stored in the memory.

[0017] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions for causing a processor to execute the target detection method provided in the embodiment of the present application.

[0018] An embodiment of the present application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, the target detection method provided in the embodiment of the present application is implemented.

[0019] The embodiments of the present application have the following beneficial effects:

[0020] In the embodiment of the present application, the trained three-dimensional detection model is used for target detection. The trained three-dimensional detection model is obtained by training the three-dimensional detection model to be trained with three-dimensional sample data, wherein the three-dimensional detection model to be trained is not an initial general model, and the first model parameter of the three-dimensional detection model to be trained is not an initial model parameter, but a model parameter obtained by fusion processing, and the first model parameter is obtained by fusing the second model parameter of the trained two-dimensional detection model, so that the three-dimensional detection model to be trained has the sequence modeling characteristics of the trained two-dimensional detection model through the fusion of model parameters, thereby improving the performance of the three-dimensional detection model to be trained. Based on this, the three-dimensional detection model to be trained is trained with three-dimensional sample data to obtain the trained three-dimensional detection model, so that the trained three-dimensional detection model has three-dimensional characteristics such as spatial position perception and geometric feature extraction on the basis of the two-dimensional characteristic of sequence modeling, so as to enrich the characteristics of the trained three-dimensional detection model, thereby improving the accuracy and efficiency of target detection. In addition, when using the trained three-dimensional detection model for target detection, it is only necessary to make predictions based on the three-dimensional data to be detected, and it is not necessary to make predictions based on two-dimensional images, so as to shorten the prediction time and improve the real-time performance of target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 1 is a schematic diagram of a network architecture of a target detection system 100 provided in an embodiment of the present application;

[0022] Figure 2 is a schematic diagram of the structure of the terminal 400 provided in an embodiment of the present application;

[0023] Figure 3 It is a first flow chart of the target detection method provided in the embodiment of the present application;

[0024] Figure 4 is a second flow chart of the target detection method provided in an embodiment of the present application;

[0025] Figure 5 It is a first flow chart of the fusion processing provided in the embodiment of the present application;

[0026] Figure 6 is a second flow chart of the fusion processing provided in the embodiment of the present application;

[0027] Figure 7 This is a flow chart of the model training provided in the embodiment of the present application;

[0028] Figure 8 is a third flow chart of the target detection method provided in an embodiment of the present application;

[0029] Fig. 9 This is a schematic diagram of the structure composition of the Transformer network structure provided in an embodiment of the present application;

[0030] Fig.10 This is a schematic diagram of the structural composition of a basic unit provided in an embodiment of the present application;

[0031] Fig.11 It is a schematic diagram of a framework structure combining weight parameters provided in an embodiment of the present application.

[0032] It should be pointed out that the above-mentioned "first" and "second" are only used to distinguish different solutions, and do not represent the degree of superiority or inferiority of the solutions or the priority in the implementation process. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0034] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0035] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0036] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0037] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0038] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and the personal information subject.

[0039] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0040] 1) 2D object detection refers to identifying and locating objects in a 2D space in an image or video. This detection usually involves identifying the type and location of the object and enclosing each detected object by drawing a bounding box.

[0041] The basic working principles of two-dimensional target detection include: data input, feature extraction, object positioning, classification, and non-maximum suppression (NMS).

[0042] Common algorithms for two-dimensional object detection include Regions with CNN features (R-CNN), Fast Regions with Convolutional Neural Networks (Fast R-CNN), Faster Regions with Convolutional Neural Networks (Faster R-CNN), You Only Look Once (YOLO), and Single Shot MultiBox Detector (SSD).

[0043] Two-dimensional object detection has a wide range of applications in scenarios such as autonomous driving, video surveillance, image processing, and human-computer interaction.

[0044] 2) 3D object detection refers to the technology of identifying and locating objects in any three-dimensional space. Unlike 2D object detection, 3D object detection not only identifies the existence and category of an object, but also determines the precise position and posture of the object in space.

[0045] Three-dimensional object detection usually includes data acquisition, preprocessing, feature extraction, detection and classification, and post-processing steps. Among them, data acquisition uses three-dimensional sensors (such as lidar, structured light sensors, depth cameras, etc.) to collect three-dimensional data of the environment. Preprocessing refers to denoising, filtering and segmenting the collected data to improve data quality. Feature extraction is to extract features that are helpful for detection from three-dimensional data. Features can include shape, size, surface texture, etc. Detection and classification is the use of algorithms to identify and classify objects in three-dimensional space, and determine the position and posture of the objects. Post-processing is to optimize the detection results, including merging overlapping detection frames, removing erroneous detections, etc.

[0046] Point cloud-based detection, voxel-based methods, multi-sensor fusion, and deep learning methods are commonly used technologies and algorithms in 3D object detection. Among them, point cloud-based detection uses algorithms directly on point cloud data, such as PointNet, PointNet++, etc. The voxel-based method converts the point cloud into a voxel grid and then applies processing techniques similar to two-dimensional images. Multi-sensor fusion combines multiple sensor data (such as LiDAR and cameras) to improve the accuracy and robustness of detection. The deep learning method uses deep learning architectures such as Convolutional Neural Networks (CNN) and Regional Convolutional Neural Networks (R-CNN) for detection.

[0047] Three-dimensional object detection is very important in scenarios such as autonomous driving, robot navigation, virtual reality (VR) and augmented reality (AR), and industrial automation.

[0048] 3) Point cloud is a collection of points in a large space, each of which contains information about its position in three-dimensional space (usually including x, y, z coordinates). These points are captured by various sensors (such as lidar, structured light sensor, depth camera, etc.) to represent the three-dimensional structure of the surface of an object or the environment.

[0049] Point cloud data is collected through lidar, structured light, depth camera, etc.

[0050] Point clouds have the characteristics of 3D coordinates, attribute information, and resolution. The 3D coordinate feature means that each point has 3D coordinates, providing its exact position in space. The attribute information feature means that in addition to coordinates, points in a point cloud may also contain other attributes, such as color, intensity, reflectivity, etc. The resolution feature means that the resolution of a point cloud depends on the sensor and acquisition settings. High resolution means that there are more points to represent an object, thus providing more detailed information.

[0051] Point cloud data usually needs to go through processing steps such as denoising, filtering, segmentation, registration, and feature extraction.

[0052] Point clouds are widely used in autonomous driving, topographic mapping, architecture and engineering, archeology, forestry and other fields.

[0053] Compared with images, point clouds provide richer three-dimensional spatial information. Images are two-dimensional and contain pixel values, which usually require deep learning models to extract features. Point clouds are three-dimensional and directly represent the geometric shape of objects, and can be used to directly measure size and distance.

[0054] In order to better understand the target detection method provided in the embodiments of the present application, the target detection method in the related art and its existing shortcomings are first described.

[0055] In related technologies, with the rapid development of artificial intelligence (AI) technology, autonomous driving has also become more and more mature. For autonomous driving, it not only includes the autonomous driving of cars on the road, but also the autonomous driving of unmanned vehicles in the park. In the process of unmanned vehicles driving, in addition to determining what kind of objects are in front, it is also necessary to determine the location of the objects. Therefore, the traditional two-dimensional target detection method cannot obtain the location of the object.

[0056] There are related technologies that only use point clouds for target detection, which can know both the category of the object in front and the location of the object, but there is a problem of low accuracy of the prediction results. There are also related technologies that use both images and point clouds for learning, and then perform post-fusion. The post-fusion includes using images to train a two-dimensional target detection model, and also using point clouds to train a three-dimensional target detection model, and then fusing the results of the three-dimensional target detection model with the two-dimensional target detection model. In addition, there are related technologies that directly fuse the features of images and point clouds at the feature layer. A relatively classic method is BEVFusion. However, the above-mentioned fusion-related technologies require two types of aligned modal data during reasoning, and both need to pass through a two-dimensional target detection model and a three-dimensional target detection model, which leads to a long reasoning time.

[0057] The unmanned logistics vehicles in the park need to know the category and category information of the obstacles ahead. In order to make full use of point cloud and image information, the relevant technology will train the images and point clouds at the same time, and will also input the images and point clouds into the network during reasoning, which will increase the time consumption.

[0058] Combining the above-mentioned related technologies, it can be seen that the disadvantages of the related technologies include at least: first, the prediction results are low in accuracy; second, the prediction takes a long time, resulting in poor real-time performance.

[0059] The embodiments of the present application provide a target detection method, apparatus, computer equipment, computer-readable storage medium, and computer program product, which can improve the efficiency of target detection. The following describes an exemplary application of the computer equipment provided by the embodiments of the present application. The computer equipment provided by the embodiments of the present application can be implemented as various types of terminals such as laptops, tablet computers, desktop computers, set-top boxes, smart phones, smart speakers, smart watches, smart TVs, unmanned vehicles, drones, robots, and aircraft, and can also be implemented as servers. Below, an exemplary application when the computer equipment is implemented as a terminal will be described.

[0060] See also Figure 1 , Figure 1 It is a schematic diagram of the network architecture of the target detection system 100 provided in an embodiment of the present application. To support a target detection application, the terminal 400 is connected to the server 200 via the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0061] The terminal 400 can be an unmanned vehicle, a drone, a robot, an aircraft, etc. The terminal 400 is used to periodically send a model acquisition request to the server 200. The server 200 is used to send a trained three-dimensional detection model to the terminal 400 based on the model acquisition request, wherein the trained three-dimensional detection model is obtained by the server 200 in advance by training the three-dimensional detection model to be trained through three-dimensional sample data, and the first model parameters of the three-dimensional detection model to be trained are obtained by fusing the second model parameters of the trained two-dimensional detection model; the first model parameters are the model parameters of the three-dimensional detection model to be trained, the trained two-dimensional detection model can perform target detection on the two-dimensional image, and the second model parameters are the model parameters of the trained two-dimensional detection model; the terminal 400 is also used to collect the three-dimensional data to be detected, and use the trained three-dimensional detection model to extract features of the three-dimensional data to be detected to obtain feature information; and predict the three-dimensional data to be detected according to the feature information to obtain the target detection result; based on this, the terminal 400 can also perform driving tasks based on the target detection result.

[0062] In the embodiment of the present application, the terminal 400 performs target detection using a trained three-dimensional detection model, which is obtained by training a three-dimensional detection model to be trained using three-dimensional sample data, wherein the three-dimensional detection model to be trained is not an initial general model, and the first model parameter of the three-dimensional detection model to be trained is not an initial model parameter, but a model parameter obtained through fusion processing, and the first model parameter is obtained by fusing the second model parameter of the trained two-dimensional detection model, so that the three-dimensional detection model to be trained has the sequence modeling characteristics of the trained two-dimensional detection model through the fusion of model parameters, thereby improving the performance of the three-dimensional detection model to be trained. Based on this, the three-dimensional detection model to be trained is trained using three-dimensional sample data to obtain a trained three-dimensional detection model, so that the trained three-dimensional detection model, on the basis of the two-dimensional characteristic of sequence modeling, also adds three-dimensional characteristics such as spatial position perception and geometric feature extraction, thereby enriching the characteristics of the trained three-dimensional detection model, thereby improving the accuracy and efficiency of target detection. In addition, when using the trained 3D detection model for target detection, it is only necessary to make predictions based on the 3D data to be detected, and there is no need to make additional predictions based on 2D images. This shortens the prediction time and improves the real-time performance of target detection.

[0063] In some embodiments, the server 200 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The terminal and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0064] See also Figure 2 , Figure 2 is a schematic diagram of the structure of the terminal 400 provided in an embodiment of the present application, Figure 2 The terminal 400 shown includes: at least one processor 410, a memory 450, at least one network interface 420 and a user interface 430. The various components in the terminal 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 440 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 440 is not shown in FIG. Figure 2 Various buses are labeled as bus system 440 .

[0065] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0066] The user interface 430 includes one or more output devices 431 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0067] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.

[0068] The memory 450 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0069] In some embodiments, memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0070] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0071] A network communication module 452, used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB), etc.;

[0072] a presentation module 453 for enabling presentation of information via one or more output devices 431 (e.g., display screen, speaker, etc.) associated with the user interface 430 (e.g., a user interface for operating peripherals and displaying content and information);

[0073] The input processing module 454 is used to detect one or more user inputs or interactions from one of the one or more input devices 432 and translate the detected inputs or interactions.

[0074] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 2 The target detection device 455 stored in the memory 450 is shown, which can be software in the form of a program and a plug-in, etc., including the following software modules: a first acquisition module 4551, a first feature extraction module 4552 and a first prediction module 4553. These modules are logical, so they can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be explained below.

[0075] In other embodiments, the device provided in the embodiments of the present application can be implemented in hardware. As an example, the device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the target detection method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application specific integrated circuits (Application Specific Integrated Circuit, ASIC), DSP, programmable logic device (Programmable Logic Device, PLD), complex programmable logic device (Complex Programmable Logic Device, CPLD), field programmable gate array (Field-Programmable Gate Array, FPGA) or other electronic components.

[0076] In some embodiments, the terminal can implement the target detection method provided in the embodiment of the present application by running various computer executable instructions or computer programs. For example, computer executable instructions can be microprogram-level commands, machine instructions or software instructions. The computer program can be a native program or software module in the operating system; it can be a native (Native) application (APPlication, APP), that is, a program that needs to be installed in the operating system to run, such as an autonomous driving APP, a target detection APP; it can also be a small program that can be embedded in any APP, that is, a program that can be run only by downloading it to a browser environment. In short, the above-mentioned computer executable instructions can be instructions in any form, and the above-mentioned computer program can be an application, module or plug-in in any form.

[0077] The target detection method provided in the embodiment of the present application will be described in conjunction with the exemplary application and implementation of the terminal provided in the embodiment of the present application.

[0078] The target detection method provided by the embodiment of the present application is described below. As mentioned above, the computer device implementing the target detection method of the embodiment of the present application can be a terminal, a server, or a combination of the two. Therefore, the execution subject of each step will not be repeatedly described below.

[0079] It should be noted that in the examples of target detection methods below, the three-dimensional data to be detected is laser radar data. Based on the understanding of the following, those skilled in the art can apply the target detection method provided in the embodiments of the present application to target detection of other three-dimensional data such as environmental mapping data, terrain analysis data, and multi-source fusion data. The embodiments of the present application can also be applied to various scenarios, including but not limited to agriculture, instant messaging, artificial intelligence, smart transportation, assisted driving, etc.

[0080] See also Figure 3 , Figure 3 is a first flow chart of the target detection method provided in the embodiment of the present application, which will be combined with Figure 3 The steps shown are used to illustrate the target detection method provided in the embodiment of the present application. Figure 3 The execution subject is the terminal.

[0081] In step S101, the three-dimensional data to be detected and the trained three-dimensional detection model are obtained.

[0082] In the embodiment of the present application, the trained 3D detection model is obtained by training the 3D detection model to be trained with 3D sample data, and the first model parameters of the 3D detection model to be trained are obtained by fusing the second model parameters of the trained 2D detection model. The first model parameters are the model parameters of the 3D detection model to be trained, the trained 2D detection model can perform target detection on 2D images, and the second model parameters are the model parameters of the trained 2D detection model.

[0083] In some embodiments, the trained two-dimensional detection model can be considered as a two-dimensional detection model that has completed training. In this way, target detection can be performed on the two-dimensional image through the trained two-dimensional detection model to obtain the detection result for the two-dimensional image. The detection result for the two-dimensional image can include at least one of the object category, object appearance shape, and object texture features.

[0084] In some embodiments, the three-dimensional data to be detected may be acquired by an acquisition device provided on the terminal, and the acquisition device may be a laser radar, a structured light, a depth camera, etc. The three-dimensional data to be detected may be a set of discrete three-dimensional space coordinate points, each of which usually has three coordinate values ​​of X, Y, and Z, which represent the position of the point in the three-dimensional space. Therefore, the three-dimensional data to be detected is essentially three-dimensional, and the three-dimensional data to be detected can express the three-dimensional structure and geometric information of the target object or scene.

[0085] In some embodiments, taking the autonomous driving scenario as an example, the trained 3D detection model refers to a model that has been trained with 3D sample data and can process 3D data in the actual environment to detect and identify the position, size and category of objects (or target objects) (such as vehicles, pedestrians, traffic signs, etc.) in the scene. The trained 3D detection model has detection capabilities such as positioning, classification, and segmentation. The trained 3D detection model also has real-time, robust and generalization capabilities, thereby providing key information for subsequent decision-making and control.

[0086] In some embodiments, before executing the above step S101, a trained 3D detection model is required. Based on this, Figure 4 is a second flow chart of the target detection method provided in the embodiment of the present application, see Figure 4 Before executing the above step S101, the following steps S001 to S004 may also be executed, which are described in detail below.

[0087] In step S001, an initial two-dimensional detection model is trained using a plurality of training sample images and first label information of the plurality of training sample images to obtain a trained two-dimensional detection model.

[0088] In some embodiments, multiple training sample images and first label information of the sample images can be first obtained from a public data set. The first label information refers to information related to each detected target in the training sample image. Exemplarily, the first label information may include target category, bounding box coordinates, center point coordinates, size information, etc.

[0089] In the embodiment of the present application, the initial two-dimensional detection model refers to the model before model training, and the trained two-dimensional detection model refers to the model obtained after training the initial two-dimensional detection model. Both the initial two-dimensional detection model and the trained two-dimensional detection model are target detection models based on two-dimensional images. Among them, the initial two-dimensional detection model can be based on a deep learning framework, such as CNN, R-CNN, Fast R-CNN, Faster R-CNN, YOLO, SSD, etc.

[0090] In some embodiments, during the training process, the training sample image and the corresponding first label information are input into the initial two-dimensional detection model. Then, a loss function is used to measure the difference between the model prediction and the true label (i.e., the first label information), wherein the loss function may be a cross entropy loss, a smoothed L1 loss, etc. Finally, the model weights are updated through a back propagation algorithm to minimize the loss function, thereby obtaining a trained two-dimensional detection model.

[0091] In some embodiments, the above step S001 can be implemented by the following steps S011 to S013, which are described in detail below.

[0092] In step S011, for each training sample image, the initial two-dimensional detection model is used to perform feature extraction and prediction processing on the training sample image to obtain a second prediction result.

[0093] In some embodiments, first, the initial two-dimensional detection model is used to extract features from the training sample image. Here, the feature extraction can be performed through the convolution layer of the initial two-dimensional detection model to obtain image features. Next, the initial two-dimensional detection model will perform prediction processing based on the image features. For the target detection task, the initial two-dimensional detection model will output a series of bounding boxes and corresponding category probabilities. Finally, a second prediction result is generated. The second prediction result is the result obtained after the initial two-dimensional detection model performs prediction processing on the training sample image. The second prediction result may include bounding box coordinates, category labels, confidence scores, etc.

[0094] In other embodiments, the sample image may be preprocessed by normalization, resizing, dimension expansion, etc. to obtain a preprocessed sample image; then, the initial two-dimensional detection model may be used to perform feature extraction and prediction processing on the preprocessed sample image to obtain a second prediction result.

[0095] In step S012, a second loss value is determined based on the first label information of the training sample image and the second prediction result.

[0096] In some embodiments, the first label information generally includes the real bounding box coordinates, category labels, etc. of the training sample image. The first label information is a benchmark during model training and is used to measure the accuracy of model prediction.

[0097] In some embodiments, when determining the second loss value, the loss function can be determined first. Here, a loss function matching the target detection task needs to be determined to measure the difference between the predicted result and the true label. Exemplary, commonly used loss functions include smooth L1 loss (Smooth L1 Loss), cross entropy loss (Cross Entropy Loss) and focal loss (Focal Loss), etc.; then, based on the loss function, the category loss, positioning loss and confidence loss are determined respectively; finally, the second loss value can be obtained by weighting the category loss, positioning loss and confidence loss.

[0098] Among them, for category loss, the category loss between the predicted category probability and the true category label can be determined by cross entropy loss. For positioning loss, the coordinate difference between the predicted bounding box and the true bounding box can be determined by smooth L1 loss. For confidence loss, it can be determined by the confidence of the initial two-dimensional detection model for the predicted bounding box and the true situation.

[0099] In step S013, the model parameters of the initial two-dimensional detection model are adjusted based on the second loss value until the training termination condition is reached, thereby obtaining a trained two-dimensional detection model.

[0100] In an embodiment of the present application, the training termination condition may be at least one of the second loss value reaching a loss value threshold, the second loss value converging, or the number of training times reaching a number threshold.

[0101] In some embodiments, the second loss value is back-propagated to adjust or update the weights of the initial two-dimensional detection model, that is, the learning and optimization of the initial two-dimensional detection model is guided based on the second loss value; and the above steps S011 to S013 are repeated until the training termination condition is reached, thereby obtaining a trained two-dimensional detection model.

[0102] Through the above steps S011 to S013, the initial two-dimensional detection model can be trained based on the training sample image and the corresponding first label information to obtain a trained two-dimensional detection model. The trained two-dimensional detection model has the characteristics of performing target detection on images, and it can also be considered that the trained two-dimensional detection model has sequence modeling capabilities.

[0103] Continue to see Figure 4 , and the description continues with step S001 above.

[0104] In step S002, based on the learnable coefficients, the initial model parameters of the initial three-dimensional detection model and the second model parameters of the trained two-dimensional detection model are fused to obtain first model parameters.

[0105] In an embodiment of the present application, the learnable coefficient is a proportional coefficient used to fuse model parameters of different detection models. The initial three-dimensional detection model and the trained two-dimensional detection model have the same network structure feature extraction unit. Based on this, the learnable coefficient can be used to fuse the unit parameters of the feature extraction unit in the trained two-dimensional detection model with the unit parameters of the feature extraction unit in the initial three-dimensional detection model to obtain a first model parameter, which can be considered as a fusion model parameter.

[0106] In some embodiments, the initial three-dimensional detection model includes a first feature extraction unit and a first detection unit, and the initial model parameters of the initial three-dimensional detection model include a first unit parameter of the first feature extraction unit and a second unit parameter of the first detection unit. Based on this, see Figure 5 The above step S002 can be implemented by the following steps S021 to S023, which are described in detail below.

[0107] In step S021, a third unit parameter of the second feature extraction unit in the trained two-dimensional detection model is obtained from the second model parameter of the trained two-dimensional detection model.

[0108] In an embodiment of the present application, the network structure of the first feature extraction unit in the initial three-dimensional detection model is the same as the network structure of the second feature extraction unit in the trained two-dimensional detection model.

[0109] In some embodiments, the trained two-dimensional detection model includes a second feature extraction unit and a second detection unit, wherein the second feature extraction unit is used to extract features to obtain image features; the second detection unit is used to detect targets based on image features. Based on this, the second model parameters include the third unit parameters of the second feature extraction and the fourth unit parameters of the second detection unit, and the third unit parameters of the second feature extraction unit can be obtained from the second model parameters.

[0110] In step S022, based on the learnable coefficient, the first unit parameter and the third unit parameter are fused to obtain the fused unit parameter.

[0111] In the embodiment of the present application, the learnable coefficient is essentially an adjustable parameter. The learnable coefficient is similar to the model parameter and is adjusted according to the loss value during the model training process. The learnable coefficient is determined when the model training is completed.

[0112] In some embodiments, the third unit parameter can be weighted by using the learnable coefficient, and then added to the first unit parameter to obtain the fused unit parameter. The third unit parameter and the first unit parameter both belong to the parameter set, based on which, each parameter in the first unit parameter can be weighted by using the learnable coefficient to obtain a weighted result; each weighted result is then added to the corresponding parameter in the first unit parameter to obtain the fused unit parameter.

[0113] In some embodiments, the learnable coefficients include weight coefficients and bias coefficients, the first unit parameters include initial weights and initial biases of multiple linear connection layers in the first feature extraction unit, and the third unit parameters include two-dimensional weights and two-dimensional biases of multiple linear connection layers in the second feature extraction unit. Based on this, see Figure 6 The above step S022 can be implemented by the following steps S221 to S225, which are described in detail below.

[0114] In step S221 , for each linear connection layer, a weighted weight is determined based on the two-dimensional weight and the weight coefficient.

[0115] In some embodiments, the product of the two-dimensional weight and the weight coefficient may be determined as the weighted weight. Since the weight coefficient is adjusted and finally determined during model training, the weight coefficient may be represented by an unknown number.

[0116] In some other embodiments, the square value of the weight coefficient may be determined first, and then the product of the square value of the weight coefficient and the two-dimensional weight may be determined as the weighted weight.

[0117] In step S222, a weighted offset is determined based on the two-dimensional offset and the offset coefficient.

[0118] Similar to the above step S221, the product of the two-dimensional offset and the offset coefficient may be determined as the weighted offset. Alternatively, the square value of the offset coefficient may be determined first, and then the product of the square value of the offset coefficient and the two-dimensional offset may be determined as the weighted offset.

[0119] In step S223, the weighted weights and the initial weights are fused to obtain the fused weights of the linear connection layer.

[0120] In some embodiments, the sum of the weighted weight and the initial weight may be determined as the fusion weight; or a first product of the weighted weight and a first preset coefficient may be determined first, and then the sum of the first product and the initial weight may be determined as the fusion weight.

[0121] In step S224, the weighted bias and the initial bias are fused to obtain a fused bias of the linear connection layer.

[0122] Similar to the above step S223, the sum of the weighted offset and the initial offset is determined as the fused offset; the second product of the weighted offset and the second preset coefficient may be determined first, and then the sum of the second product and the initial offset is determined as the fused offset.

[0123] In step S225, each fusion weight and each fusion bias are determined as the fused unit parameters of the first feature extraction unit.

[0124] In the embodiment of the present application, the fused unit parameters correspond to the first unit parameters, that is, the fused unit parameters correspond to the multiple linear connection layers in the first feature extraction unit.

[0125] Through the above steps S221 to S225, the initial weights and the two-dimensional weights can be fused in a weighted manner, and the initial bias and the two-dimensional bias can be fused in a weighted manner to obtain the fused weights and the fused bias, that is, the fused unit parameters, thereby realizing the parameter fusion of the feature extraction units in the two models.

[0126] Continue to see Figure 5 , and the description continues with step S022 above.

[0127] In step S023, the fused unit parameters and the second unit parameters are determined as first model parameters.

[0128] In an embodiment of the present application, the fused unit parameters are the parameters corresponding to the first feature extraction unit in the initial three-dimensional detection model, and the second unit parameters are the parameters corresponding to the first detection unit in the initial three-dimensional detection model, so the fused unit parameters and the second unit parameters are determined as the first model parameters.

[0129] Through the above steps S021 to S023, since the network structure of the first feature extraction unit in the initial three-dimensional detection model is the same as the network structure of the second feature extraction unit in the trained two-dimensional detection model, the fusion of the first unit parameter and the third unit parameter can be achieved in a weighted manner, and the fused unit parameter of the first feature extraction unit is obtained. In addition, the second unit parameter of the first detection unit in the initial three-dimensional detection model is kept unchanged, and finally the fused unit parameter and the second unit parameter are jointly determined as the first model parameter, thereby realizing the fusion of the two-dimensional network feature and the three-dimensional network feature.

[0130] Continue to see Figure 4 , and the description continues with step S002 above.

[0131] In step S003, the initial model parameters in the initial three-dimensional detection model are updated using the first model parameters to obtain a three-dimensional detection model to be trained.

[0132] In the embodiment of the present application, the initial model parameters in the initial 3D modeling are replaced with the first model parameters, that is, the initial model parameters in the initial 3D modeling are adjusted to obtain an adjusted 3D detection model, which is recorded as the 3D detection model to be trained. The full adjustment adopted here can ensure the consistency, comprehensiveness and accuracy of the update.

[0133] In some embodiments, the fusion unit parameters in the first model parameters can also be used to update the first unit parameters in the initial model parameters, thereby adjusting the initial model parameters in the initial three-dimensional model building to obtain the three-dimensional detection model to be trained. What is performed here is local adjustment rather than full adjustment, which can shorten the update time and improve the efficiency and flexibility of the update.

[0134] In step S004, the three-dimensional detection model to be trained is trained using the plurality of three-dimensional sample data and the second label information of the plurality of three-dimensional sample data to obtain a trained three-dimensional detection model.

[0135] In some embodiments, the learnable coefficient is iteratively updated during the training of the three-dimensional detection model to be trained, so that during the training of the three-dimensional detection model to be trained, the iteratively updated parameters include the learnable coefficient and the first model parameter in the three-dimensional detection model to be trained. A plurality of three-dimensional sample data and second label information of the three-dimensional sample data can be first obtained from a public data set, and the second label information refers to information related to each detection target in the three-dimensional sample data. Exemplarily, the second label information may include position information, texture information, depth information, etc.

[0136] In some embodiments, the three-dimensional sample data may be first cleaned and preprocessed to obtain processed data; the processed data may then be input into the three-dimensional detection model to be trained, and the trained three-dimensional detection model may be obtained through forward propagation, determination of the first loss value, back propagation, and iterative optimization. Among them, data cleaning includes removing noise, outliers, and irrelevant three-dimensional sample data; data preprocessing includes standardizing the three-dimensional sample data, exemplarily normalizing coordinates, so as to ensure the consistency of the input data; the first loss value is determined based on the first prediction result obtained by the forward propagation and the second label information.

[0137] In some embodiments, see Figure 7 The above step S004 can be implemented by the following steps S041 to S043, which are described in detail below.

[0138] In step S041, for each three-dimensional sample data, the three-dimensional detection model to be trained is used to perform feature extraction and prediction processing on the three-dimensional sample data to obtain a first prediction result.

[0139] In an embodiment of the present application, first, three-dimensional sample data is collected, and the three-dimensional sample data may include at least one of point cloud data, voxel data, or three-dimensional model data. Then, the three-dimensional sample data is preprocessed, such as denoising, normalization, segmentation, etc., to obtain preprocessed three-dimensional sample data to ensure the quality of the three-dimensional sample data. Next, the preprocessed three-dimensional sample data is input into the three-dimensional detection model to be trained, and the three-dimensional detection model to be trained analyzes the preprocessed three-dimensional sample data through the encoder part (such as the feature learning layer of PointNet or the convolution layer of VoxelNet) to extract three-dimensional features; wherein the purpose of feature extraction is to capture key information in the preprocessed three-dimensional sample data, such as the shape, position, direction, etc. of the target object. Finally, using the extracted three-dimensional features, the decoder part (or classification / regression layer) of the three-dimensional detection model to be trained is used for prediction processing, and the prediction processing may include classification tasks (such as identifying the category of the target object) or regression tasks (such as predicting the position and size of the target object), and the prediction processing may also include a combination of classification tasks and regression tasks. And obtain a first prediction result, and obtain the first prediction result according to the prediction output of the three-dimensional detection model to be trained; wherein the first prediction result may include: the category of the target object, the bounding box of the target object, the direction or other attributes of the target object. Exemplarily, the category of the target object may be a vehicle, a pedestrian, etc.; the bounding box of the target object is used to locate the position and size of the target in the three-dimensional space; other attributes of the target object may be speed, acceleration, etc.

[0140] In other embodiments, the implementation process of step S041 is similar to the implementation process of step S011, except that step S041 uses three-dimensional sample data and the trained model is a three-dimensional detection model to be trained, while step S011 uses training sample images and the trained model is an initial two-dimensional detection model. Based on this, the implementation process of step S041 can refer to the implementation process of step S011.

[0141] In step S042, a first loss value is determined based on the second label information of the three-dimensional sample data and the first prediction result.

[0142] In some embodiments, the second label information is real annotation information related to the three-dimensional sample data, and the second label information may include: at least one of the category label of the target object, the location of the target object (such as three-dimensional coordinates, bounding box), and other attributes of the target object (such as rotation angle, speed, etc.). The first prediction result is the prediction value output by the three-dimensional detection model to be trained after processing the three-dimensional sample data. The first prediction result corresponds to the second label information, that is, the first prediction result includes: at least one of the predicted target object category, the predicted target object location and size, and other predicted attributes.

[0143] In other embodiments, if the three-dimensional detection model to be trained outputs a predicted target object category, a classification loss function is used to determine the classification loss between the predicted target object category and the category label of the target object; if the three-dimensional detection model to be trained outputs predicted other attributes, a regression loss function is used to determine the regression loss between the predicted other attributes and other attributes of the target object. Then, the weighted sum of the classification loss and the regression loss is determined as the first loss value; or the classification loss, the regression loss, and the weighted sum of the classification loss and the regression loss are accumulated to obtain the first loss value.

[0144] In step S043, the first model parameters and the learnable coefficients are adjusted based on the first loss value, and the second model parameters in the first model parameters are kept unchanged until the training termination condition is reached, thereby obtaining a trained three-dimensional detection model.

[0145] In an embodiment of the present application, the first loss value is back-propagated, the first loss value is assigned to the first model parameters and the learnable coefficients in the three-dimensional detection model to be trained, and the gradients of the first model parameters and the learnable coefficients are determined, that is, the learnable coefficients are iteratively updated during the training process of the three-dimensional detection model to be trained; then the gradient descent method or other optimization algorithms are used to update the first model parameters and the learnable coefficients to reduce the first loss value; and the above steps S041 to S043 are repeated until the training termination condition is reached, thereby obtaining the trained three-dimensional detection model.

[0146] In some embodiments, the training termination condition may be at least one of the first loss value reaching a loss value threshold, the first loss value converging, or the number of training times reaching a number threshold.

[0147] Through the above steps S041 to S043, in the process of training the 3D detection model to be trained, the first model parameters and the learnable coefficients are adjusted, and the second model parameters are kept unchanged, that is, the fused unit parameters are kept unchanged, so that when the training is completed, the trained 3D detection model also has the characteristics of 2D target detection, that is, the trained 3D detection model not only has the characteristics of 3D target detection, but also has the characteristics of 2D target detection, thereby improving the performance of the trained 3D detection model.

[0148] Through the above steps S001 to S004, the trained two-dimensional detection model is first obtained through training; and after fusing the first unit parameter of the initial three-dimensional detection model and the third unit parameter of the trained two-dimensional detection model, the obtained three-dimensional detection model to be trained is trained again, so as to obtain a trained three-dimensional detection model with two-dimensional target detection characteristics and three-dimensional target detection characteristics, thereby improving the performance of the trained three-dimensional detection model.

[0149] Continue to see Figure 3 , and the description continues with step S101 above.

[0150] In step S102, the trained three-dimensional detection model is used to extract features from the three-dimensional data to be detected to obtain feature information.

[0151] In some embodiments, the implementation process of step S102 is similar to the feature extraction process of step S041 above. Therefore, the implementation process of step S102 may refer to the feature extraction process of step S041 above.

[0152] In step S103, prediction processing is performed on the three-dimensional data to be detected according to the feature information to obtain a target detection result.

[0153] In some embodiments, the implementation process of step S103 is similar to the prediction process of step S041, and therefore, the implementation process of step S103 can refer to the prediction process of step S041. The difference is that step S041 uses the 3D detection model to be trained, and step S041 predicts the 3D sample data; while step S103 uses the trained 3D detection model, and step S103 predicts the 3D data to be detected.

[0154] In some embodiments, the target detection result may include object category and object location information, and the target detection result may also include at least one of the object appearance shape, object appearance color, and object texture features.

[0155] Through the above steps S101 to S103, the trained three-dimensional detection model is used to perform target detection. The trained three-dimensional detection model is obtained by training the three-dimensional detection model to be trained with three-dimensional sample data, wherein the three-dimensional detection model to be trained is not an initial general model, and the first model parameter of the three-dimensional detection model to be trained is not an initial model parameter, but a model parameter obtained through fusion processing, and the fusion unit parameter in the first model parameter is obtained by fusing the first unit parameter and the second unit parameter, so that the three-dimensional detection model to be trained has the sequence modeling characteristics of the trained two-dimensional detection model through the parameter fusion of the feature extraction unit, thereby improving the performance of the three-dimensional detection model to be trained. Based on this, the three-dimensional detection model to be trained is trained with three-dimensional sample data to obtain the trained three-dimensional detection model, so that the trained three-dimensional detection model has added three-dimensional characteristics such as spatial position perception and geometric feature extraction on the basis of the two-dimensional characteristic of sequence modeling, thereby enriching the characteristics of the trained three-dimensional detection model, thereby improving the accuracy and efficiency of target detection. In addition, when using the trained 3D detection model for target detection, it is only necessary to make predictions based on the 3D data to be detected, and there is no need to make additional predictions based on 2D images. This shortens the prediction time and improves the real-time performance of target detection.

[0156] In some embodiments, for the field of autonomous driving, Figure 8 is a third flow chart of the target detection method provided in the embodiment of the present application, see Figure 8 After the above step S103, the following steps S104 to S106 may also be performed, which are described in detail below.

[0157] In step S104, the moving path of the electronic device is obtained.

[0158] In the embodiment of the present application, the electronic device may be an autonomous driving or automatically moving device such as an unmanned vehicle, a drone, a robot, etc. The moving path refers to the moving path currently executed by the electronic device, and the electronic device can move to the target location based on the moving path.

[0159] In step S105, when it is determined based on the target detection result that the moving path meets the update condition, the moving path is updated based on the target detection result to obtain an updated moving path.

[0160] In an embodiment of the present application, when the electronic device moves from a starting position to a target position based on a moving path, taking into account the uncertainty and variability of the environment in which the moving path is located, if the target detection result indicates that there is a preset obstacle type within a range less than a first distance threshold from the moving path, it indicates that the mobile device cannot move to the target position according to the preset path. In order for the electronic device to move to the target position smoothly, it is necessary to update the moving path based on the target detection result to obtain an updated moving path.

[0161] In some embodiments, the moving path includes multiple path coordinates, and the target detection result includes the object type of the target object represented by the three-dimensional data to be detected and the position information of the target object. Based on this, whether the moving path meets the update condition can be determined in the following way:

[0162] When the object type is a preset obstacle type, and based on the location information and multiple path coordinates, it is determined that the target object is less than a preset first distance threshold from the moving path, the first distance between the target object and the electronic device is determined; when the first distance is less than a preset second distance threshold, it is determined that the moving path meets the update condition.

[0163] Here, the preset obstacle type is a type set in advance. Exemplarily, the preset obstacle type may be a biological body, a building, a vehicle, a traffic control device, etc. The preset obstacle type represents the type that hinders the movement of the electronic device.

[0164] In an embodiment of the present application, when the object type of the target object is an obstacle type, that is, when the target object is an obstacle, the second distance between the target object and each path coordinate is determined based on the position information of the target object and each path coordinate, and the minimum second distance is determined as the third distance between the target object and the moving path; then, the magnitude relationship between the third distance and the first distance threshold is determined. If the third distance is less than the first distance threshold, it indicates that the electronic device will collide or scratch the target object if it moves according to the moving path. At this time, the first distance between the target object and the electronic device is determined based on the current position of the electronic device and the position information of the target object; finally, the magnitude relationship between the first distance and the preset second distance threshold is determined. If the first distance is less than the second distance threshold, it indicates that the electronic device is about to move to the location of the target object, and at this time, it is determined that the moving path meets the update condition. That is, when the object type, the second distance and the first distance all meet the conditions, it is determined that the moving path meets the update condition.

[0165] In some embodiments, the preset first distance threshold and the preset second distance threshold are both values ​​set in advance, and the preset first distance threshold and the preset second distance threshold may be the same or different. And the preset first distance threshold and the preset second distance threshold are both values ​​corresponding to the electronic device, that is, different electronic devices may have different first distance thresholds and second distance thresholds. Exemplarily, the preset first distance threshold may be 1 meter, 1.2 meters, 2 meters, etc., and the preset second distance threshold may be 1 meter, 2 meters, 5 meters, etc.

[0166] In other embodiments, when the object type is not a preset obstacle type, or when it is determined based on the location information and multiple path coordinates that the distance between the target object and the moving path is greater than a preset first distance threshold, or when the first distance between the target object and the electronic device is greater than a preset second distance threshold, it is determined that the moving path does not meet the update condition. That is, when at least one of the object type, the second distance, and the first distance does not meet the condition, it is determined that the moving path does not meet the update condition.

[0167] When the object type is not the preset obstacle type, the target object may be non-obstacles such as water, leaves, or paper, and there is no need to update the moving path; when the third distance is greater than the preset first distance threshold, the target object is outside the driving area of ​​the electronic device, and when the position of the target object does not change and the electronic device moves along the preset path, the electronic device will not collide or scratch the target object, and there is no need to update the moving path; when the first distance is greater than the preset second distance threshold, the electronic device is far away from the target object, and before the first distance is equal to or less than the preset second distance threshold, the target object may have been away from the moving path, that is, the third distance may have been greater than the preset first distance threshold, and there is no need to update the moving path.

[0168] In some embodiments, if the moving path does not satisfy the update condition, the electronic device is controlled to continue moving according to the moving path without updating the moving path.

[0169] In step S106, the electronic device is controlled to move according to the updated moving path.

[0170] In some embodiments, the updated movement path includes updated path coordinates, and the electronic device moves according to the updated path coordinates, thereby moving toward the target location.

[0171] Through the above steps S104 to S106, the electronic device can be guided to move in real time and accurately based on the target detection result, thereby improving the safety and intelligence of the electronic device.

[0172] The following is an explanation of an exemplary application of the embodiments of the present application in a practical application scenario.

[0173] The embodiment of the present application proposes a target detection method, and the target detection method of the embodiment of the present application can use unaligned cross-modal data to enhance the perception ability of the park logistics unmanned vehicle. That is, when using the same Transformer-based network structure, first train a two-dimensional (2Dimensional, 2D) image target detection model, and then train a three-dimensional (3Dimensions, 3D) target detection model, the linear connection module in the feedforward neural network (Feed Forward Neural Networks, FFN) module in the trained 2D image target detection model. Combine the weights with the linear connection module in the FFN module in the 3D target detection model, so as to achieve the mapping of the sequence modeling learned by the Transformer in the 2D image to the 3D target detection. Among them, the 2D image target detection model corresponds to the initial two-dimensional detection model in other embodiments, the trained 2D image target detection model corresponds to the trained two-dimensional detection model in other embodiments, and the 3D target detection model corresponds to the initial three-dimensional detection model in other embodiments.

[0174] The embodiment of the present application uses only point cloud data during reasoning, and uses the same network to train the 2D image target detection model during training, and then combines the weights of the trained 2D image target detection model with the 3D target detection model, and then trains the combined 3D target detection model, thereby enhancing the performance of the point cloud network, that is, enhancing the sequence modeling capability of the 3D target detection model. Among them, the point cloud network is the 3D target detection model. The weights of the combined 3D target detection model correspond to the three-dimensional detection model to be trained in other embodiments.

[0175] Fig. 9 This is a schematic diagram of the structure of the Transformer network structure provided in the embodiment of the present application, see Fig. 9 The Transformer network structure includes an encoder 901 and a decoder 902, wherein the encoder 901 includes N basic units and the decoder 902 also includes N basic units. Each basic unit in the encoder 901 includes a multi-head attention layer 9011, a layer normalization 9012, a pre-feedback layer 9013, and a layer normalization 9014; each basic unit in the decoder 902 includes a masked multi-head attention layer 9021, a layer normalization 9022, a multi-head attention layer 9023, a layer normalization 9024, a pre-feedback layer 9025, and a layer normalization 9026.

[0176] Fig.10This is a schematic diagram of the structure of the basic unit provided in the embodiment of the present application, see Fig.10 , the basic unit includes regularization 1001, attention mechanism 1002, mapping 1003, regularization 1004, fully connected layer 1005 and fully connected layer 1006. Both fully connected layer 1005 and fully connected layer 1006 are composed of linear connection layer and activation function. The linear connection layer can be expressed by the following formula (1):

[0177] y=ωx+b Formula (1);

[0178] In formula (1), ω represents weight and b represents bias.

[0179] When training a 3D target detection model, first use the same network structure and the collected 2D images to train the 2D image target detection model, and then train the 3D target detection model. Before training the 3D target detection model, it is necessary to combine the weight parameters of the corresponding places in the trained 2D image target detection model and the 3D target detection model.

[0180] In some embodiments, the weight parameters are combined with schematic diagrams such as Fig.11 As shown, in Fig.11 In the figure, 1101 is the point cloud part, which is the basic unit of the 3D target detection model; 1102 is the image part, which is the basic unit of the 2D target detection model. When training the point cloud part 1101, the parameters of the image part 1102 are kept fixed, that is, the 2D image target detection model does not change with the training of the point cloud part after training. The parameters of the image part 1102 are Fig.11 W′ and b′ in the fully connected layer (FC) layer. Since there is a linear part in the fully connected layer (FC) layer, the weight of the corresponding layer in the image layer is added to the weight of the 3D target detection model being trained at a certain ratio λ during training. In this way, when training the 3D target detection model, λ will also be updated during network optimization. When the training is completed, the weights of the FC parts of the two networks are added together as the weights of the FC part of the final 3D target detection model. At the same time, the weights of other parts of the 3D target detection model still retain the weights of the point cloud part training. Among them, λ corresponds to the learnable coefficient in other embodiments, and the final 3D target detection model corresponds to the trained three-dimensional detection model in other embodiments.

[0181] By using the method provided in the embodiment of the present application, both image information and 3D point cloud information are used in the model training process. The Transformer network trained with 2D images already has the ability to model image sequences. When training 3D point clouds, this part is added to assist in training, and the sequence modeling capability can be passed on to the newly trained model, which can effectively improve the 3D target detection capability.

[0182] It can be understood that in the embodiments of the present application, the collection, use and processing of relevant data such as three-dimensional data to be detected, training sample images, first label information, three-dimensional sample data, second label information and movement path must comply with relevant laws, regulations and standards.

[0183] The following further describes an exemplary structure of the target detection device 455 provided in the embodiment of the present application implemented as a software module. In some embodiments, for example Figure 2 As shown, the software modules stored in the target detection device 455 of the memory 450 may include:

[0184] A first acquisition module 4551 is used to acquire the three-dimensional data to be detected and the trained three-dimensional detection model, wherein the trained three-dimensional detection model is obtained by training the three-dimensional detection model to be trained with the three-dimensional sample data, and the first model parameters of the three-dimensional detection model to be trained are obtained by fusing the second model parameters of the trained two-dimensional detection model; wherein the first model parameters are the model parameters of the three-dimensional detection model to be trained, the trained two-dimensional detection model can perform target detection on the two-dimensional image, and the second model parameters are the model parameters of the trained two-dimensional detection model;

[0185] A first feature extraction module 4552 is used to extract features from the three-dimensional data to be detected using the trained three-dimensional detection model to obtain feature information;

[0186] The first prediction module 4553 is used to perform prediction processing on the three-dimensional data to be detected according to the feature information to obtain a target detection result.

[0187] In some embodiments, the software modules stored in the object detection device 455 of the memory 450 may also include:

[0188] A first training module, used to train an initial two-dimensional detection model using a plurality of training sample images and a plurality of first label information of the training sample images to obtain a trained two-dimensional detection model;

[0189] A fusion module, used for fusing the initial model parameters of the initial three-dimensional detection model and the second model parameters of the trained two-dimensional detection model based on a learnable coefficient to obtain a first model parameter; the learnable coefficient is a proportional coefficient used to fuse the model parameters of different detection models;

[0190] A first updating module, used for updating initial model parameters in the initial three-dimensional detection model using the first model parameters to obtain a three-dimensional detection model to be trained;

[0191] The second training module is used to train the three-dimensional detection model to be trained by using multiple three-dimensional sample data and multiple second label information of the three-dimensional sample data to obtain the trained three-dimensional detection model, wherein the learnable coefficient is iteratively updated during the training process of the three-dimensional detection model to be trained.

[0192] In some embodiments, the initial three-dimensional detection model includes a first feature extraction unit and a first detection unit, the initial model parameters of the initial three-dimensional detection model include a first unit parameter of the first feature extraction unit and a second unit parameter of the first detection unit, and the fusion module is further used to:

[0193] From the second model parameters of the trained two-dimensional detection model, obtain the third unit parameters of the second feature extraction unit in the trained two-dimensional detection model, the network structure of the first feature extraction unit in the initial three-dimensional detection model is the same as the network structure of the second feature extraction unit in the trained two-dimensional detection model; based on the learnable coefficient, the first unit parameters and the third unit parameters are fused to obtain fused unit parameters; the fused unit parameters and the second unit parameters are determined as the first model parameters.

[0194] In some embodiments, the learnable coefficient includes a weight coefficient and a bias coefficient, the first unit parameter includes initial weights and initial biases of multiple linear connection layers in the first feature extraction unit, and the third unit parameter includes two-dimensional weights and two-dimensional biases of multiple linear connection layers in the second feature extraction unit; the fusion module is further used to:

[0195] For each linear connection layer, a weighted weight is determined based on the two-dimensional weight and the weight coefficient; a weighted bias is determined based on the two-dimensional bias and the bias coefficient; the weighted weight and the initial weight are fused to obtain a fused weight of the linear connection layer; the weighted bias and the initial bias are fused to obtain a fused bias of the linear connection layer; each fused weight and each fused bias are determined as a fused unit parameter of the first feature extraction unit.

[0196] In some embodiments, the second training module is further configured to:

[0197] For each of the three-dimensional sample data, the three-dimensional detection model to be trained is used to perform feature extraction and prediction processing on the three-dimensional sample data to obtain a first prediction result; based on the second label information of the three-dimensional sample data and the first prediction result, a first loss value is determined; based on the first loss value, the first model parameters and the learnable coefficient are adjusted, and the second model parameters in the first model parameters are kept unchanged until the training termination condition is reached, so as to obtain the trained three-dimensional detection model.

[0198] In some embodiments, the first training module is further configured to:

[0199] For each of the training sample images, the initial two-dimensional detection model is used to perform feature extraction and prediction processing on the training sample image to obtain a second prediction result; based on the first label information of the training sample image and the second prediction result, a second loss value is determined; based on the second loss value, the model parameters of the initial two-dimensional detection model are adjusted until the training termination condition is reached, thereby obtaining the trained two-dimensional detection model.

[0200] In some embodiments, the software module in the target detection device 455 may also include:

[0201] A second acquisition module, used to acquire a moving path of the electronic device;

[0202] A second updating module, configured to update the moving path based on the target detection result to obtain an updated moving path when it is determined based on the target detection result that the moving path meets an update condition;

[0203] The moving module is used to control the electronic device to move according to the updated moving path.

[0204] In some embodiments, the moving path includes a plurality of path coordinates, the target detection result includes the object type of the target object represented by the three-dimensional data to be detected and the position information of the target object, and the software module in the target detection device 455 may further include:

[0205] A first determination module, configured to determine a first distance between the target object and the electronic device when the object type is a preset obstacle type and it is determined based on the position information and the plurality of path coordinates that the distance between the target object and the moving path is less than a preset first distance threshold;

[0206] The second determining module is used to determine that the moving path meets the update condition when the first distance is less than a preset second distance threshold.

[0207] The embodiment of the present application provides a computer program product or a computer program, which includes computer executable instructions, and the computer executable instructions are stored in a computer readable storage medium. The processor of the computer device reads the computer executable instructions from the computer readable storage medium, and the processor executes the computer executable instructions, so that the computer device performs the target detection method described in the embodiment of the present application.

[0208] The present application embodiment provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, the processor will be caused to execute the target detection method provided by the present application embodiment, for example, Figure 3 , 4 , 8 shows the target detection method.

[0209] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.

[0210] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0211] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0212] By way of example, computer executable instructions may be deployed to be executed on one computing device or on multiple computing devices located at one site or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0213] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A target detection method, characterized in that: The method comprises: Acquire three-dimensional data to be detected and a trained three-dimensional detection model, wherein the trained three-dimensional detection model is obtained by training the three-dimensional detection model to be trained with three-dimensional sample data, and first model parameters of the three-dimensional detection model to be trained are obtained by fusing second model parameters of a trained two-dimensional detection model; wherein the first model parameters are model parameters of the three-dimensional detection model to be trained, the trained two-dimensional detection model can perform target detection on a two-dimensional image, and the second model parameters are model parameters of the trained two-dimensional detection model; Using the trained three-dimensional detection model to extract features from the three-dimensional data to be detected to obtain feature information; The three-dimensional data to be detected is predicted and processed according to the feature information to obtain a target detection result.

2. The method according to claim 1, characterized in that Before obtaining the trained three-dimensional detection model, the method further includes: Using a plurality of training sample images and a plurality of first label information of the training sample images, an initial two-dimensional detection model is trained to obtain a trained two-dimensional detection model; Based on the learnable coefficient, the initial model parameters of the initial three-dimensional detection model and the second model parameters of the trained two-dimensional detection model are fused to obtain the first model parameters; the learnable coefficient is a proportional coefficient used to fuse the model parameters of different detection models; Using the first model parameters to update the initial model parameters in the initial three-dimensional detection model to obtain a three-dimensional detection model to be trained; The three-dimensional detection model to be trained is trained using multiple three-dimensional sample data and multiple second label information of the three-dimensional sample data to obtain the trained three-dimensional detection model, wherein the learnable coefficient is iteratively updated during the training process of the three-dimensional detection model to be trained.

3. The method according to claim 2, characterized in that The initial three-dimensional detection model includes a first feature extraction unit and a first detection unit, the initial model parameters of the initial three-dimensional detection model include a first unit parameter of the first feature extraction unit and a second unit parameter of the first detection unit, and the initial model parameters of the initial three-dimensional detection model and the second model parameters of the trained two-dimensional detection model are fused based on the learnable coefficient to obtain the first model parameters, including: Obtaining a third unit parameter of a second feature extraction unit in the trained two-dimensional detection model from the second model parameter of the trained two-dimensional detection model, wherein the network structure of the first feature extraction unit in the initial three-dimensional detection model is the same as the network structure of the second feature extraction unit in the trained two-dimensional detection model; Based on the learnable coefficient, the first unit parameter and the third unit parameter are fused to obtain fused unit parameters; The fused unit parameters and the second unit parameters are determined as the first model parameters.

4. The method according to claim 3, characterized in that The learnable coefficient includes a weight coefficient and a bias coefficient, the first unit parameter includes initial weights and initial biases of multiple linear connection layers in the first feature extraction unit, and the third unit parameter includes two-dimensional weights and two-dimensional biases of multiple linear connection layers in the second feature extraction unit; based on the learnable coefficient, the first unit parameter and the third unit parameter are fused to obtain the fused unit parameter, including: For each linear connection layer, determining a weighted weight based on the two-dimensional weight and the weight coefficient; Determining a weighted bias based on the two-dimensional bias and the bias coefficient; Fusing the weighted weight and the initial weight to obtain a fusion weight of the linear connection layer; Fusing the weighted bias and the initial bias to obtain a fused bias of the linear connection layer; The respective fusion weights and the respective fusion biases are determined as fused unit parameters of the first feature extraction unit.

5. The method according to claim 2, characterized in that: The method of using a plurality of three-dimensional sample data and a plurality of second label information of the three-dimensional sample data to train the three-dimensional detection model to be trained to obtain the trained three-dimensional detection model includes: For each of the three-dimensional sample data, use the three-dimensional detection model to be trained to perform feature extraction and prediction processing on the three-dimensional sample data to obtain a first prediction result; Determining a first loss value based on the second label information of the three-dimensional sample data and the first prediction result; The first model parameter and the learnable coefficient are adjusted based on the first loss value, and the second model parameter in the first model parameter is kept unchanged until the training termination condition is reached, so as to obtain the trained three-dimensional detection model.

6. The method according to claim 2, characterized in that The method of training the initial two-dimensional detection model by using a plurality of training sample images and the first label information of the plurality of training sample images to obtain a trained two-dimensional detection model includes: For each of the training sample images, use the initial two-dimensional detection model to perform feature extraction and prediction processing on the training sample image to obtain a second prediction result; Determining a second loss value based on the first label information of the training sample image and the second prediction result; The model parameters of the initial two-dimensional detection model are adjusted based on the second loss value until the training termination condition is reached to obtain the trained two-dimensional detection model.

7. The method according to any one of claims 1 to 6, characterized in that: After obtaining the target detection result, the method further includes: Obtaining a moving path of the electronic device; When it is determined based on the target detection result that the moving path satisfies an update condition, updating the moving path based on the target detection result to obtain an updated moving path; The electronic device is controlled to move according to the updated moving path.

8. The method according to claim 7, characterized in that The moving path includes a plurality of path coordinates, the target detection result includes the object type of the target object represented by the to-be-detected three-dimensional data and the position information of the target object, and the method further includes: When the object type is a preset obstacle type and it is determined based on the position information and the plurality of path coordinates that the distance between the target object and the moving path is less than a preset first distance threshold, determining a first distance between the target object and the electronic device; When the first distance is less than a preset second distance threshold, it is determined that the moving path meets the update condition.

9. A target detection device, characterized in that: The device comprises: A first acquisition module is used to acquire the three-dimensional data to be detected and the trained three-dimensional detection model, wherein the trained three-dimensional detection model is obtained by training the three-dimensional detection model to be trained with the three-dimensional sample data, and the first model parameters of the three-dimensional detection model to be trained are obtained by fusing the second model parameters of the trained two-dimensional detection model; wherein the first model parameters are the model parameters of the three-dimensional detection model to be trained, the trained two-dimensional detection model can perform target detection on the two-dimensional image, and the second model parameters are the model parameters of the trained two-dimensional detection model; A first feature extraction module, used to extract features from the three-dimensional data to be detected using the trained three-dimensional detection model to obtain feature information; The first prediction module is used to perform prediction processing on the three-dimensional data to be detected according to the feature information to obtain a target detection result.

10. A computer device, characterized in that: The computer device comprises: A memory for storing computer executable instructions; A processor, configured to implement the target detection method according to any one of claims 1 to 8 when executing the computer executable instructions stored in the memory.

11. A computer-readable storage medium storing computer-executable instructions, characterized in that: When the computer executable instructions are executed by a processor, the target detection method according to any one of claims 1 to 8 is implemented.

12. A computer program product comprising a computer program or computer executable instructions, characterized in that: When the computer program or computer executable instructions are executed by a processor, the target detection method according to any one of claims 1 to 8 is implemented.

Images