Target detection network model training method based on pixel and point cloud feature fusion

Abstract

The embodiment of the application relates to a target detection network model training method based on pixel and point cloud feature fusion, comprising the following steps: acquiring a point cloud and an image of a driving environment of an engineering vehicle; projecting the point cloud onto the image to obtain a projection image; acquiring pixel channel features of the projection image; training an initial network model based on the pixel channel features to obtain a trained target detection network model. According to the method, the point cloud is bound with different numbers of pixel channels to obtain the projection image, and then the target detection network model is trained according to the projection image and the image features corresponding to the projection image, so that the target detection network model trained in this way can avoid errors caused by relative motion of sensors, and the target detection is more accurate.

Description

Training Method for Object Detection Network Model Based on Pixel and Point Cloud Feature Fusion Technical Field

[0001] This invention relates to the field of autonomous driving, and in particular to a method for training a target detection network model based on the fusion of pixel and point cloud features. Background Technology

[0002] LiDAR, as a sensor, uses light pulses to measure the 3D coordinates of targets in a scene. However, it suffers from drawbacks such as sparsity and limited range; that is, the farther away from the sensor, the fewer points are returned. This means that distant targets may only yield a few points, or none at all, and may not be captured by the LiDAR sensor alone, making it difficult to distinguish target categories, as shown in Figure 1. Meanwhile, image input from vehicle-mounted cameras is very dense, which is beneficial for semantic understanding tasks such as detection and target segmentation. With its high resolution, the camera can detect distant targets very effectively, but it is less accurate in measuring distance; therefore, point cloud and image fusion algorithms are widely used.

[0003] However, point cloud and image fusion presents numerous challenges in engineering applications. Firstly, the vibration of the vehicle during movement causes relative motion between the two sensors, leading to changes in extrinsic parameters and vibration errors. Secondly, inherent limitations in the laser algorithm cause the laser points to fluctuate, and camera intrinsic parameter calibration suffers from minor errors due to environmental factors such as lighting, resulting in inherent calibration errors in the jointly calibrated extrinsic parameters. Furthermore, differences in clock oscillation and frequency between the sensors cause temporal and spatial discrepancies in the data acquired by the two sensors at the same time, meaning the algorithm's input data is not perfectly matched from the outset, exhibiting varying degrees of deviation due to various reasons. Summary of the Invention

[0004] In view of this, in order to solve the above-mentioned technical problems or some of the technical problems, the present invention provides a method for training a target detection network model based on the fusion of pixel and point cloud features.

[0005] In a first aspect, embodiments of the present invention provide a method for training a target detection network model based on the fusion of pixel and point cloud features, comprising:

[0006] Acquire point cloud and images of the driving environment of the engineering vehicle;

[0007] The point cloud is projected onto the image to obtain a projected image;

[0008] Obtain the pixel channel features of the projected image;

[0009] The initial network model is trained based on the pixel channel features to obtain the trained target detection network model.

[0010] In one possible implementation, the method further includes:

[0011] When errors exist, the point cloud is bound to the pixel channels of the image based on preset rules to obtain a projected image.

[0012] In one possible implementation, the method further includes:

[0013] When only horizontal or vertical errors exist, control each point cloud to bind a first preset number of pixel channels;

[0014] When both the lateral and longitudinal errors exist simultaneously, control each point cloud to bind a second preset number of pixel channels;

[0015] The image after the point cloud and pixel channels are bound together is used as the projected image.

[0016] In one possible implementation, the method further includes:

[0017] An image semantic segmentation method is used to obtain the category of the pixel channel bound to each point cloud;

[0018] Based on the category, determine the pixel channel features of the pixel region composed of pixel channels bound to each point cloud.

[0019] In one possible implementation, the method further includes:

[0020] The pixel channel features are input into a multilayer perceptron for feature mixing to obtain pixel channel mixed features.

[0021] In one possible implementation, the method further includes:

[0022] The pixel channel blending features are input into the initial network model for model training until the output of the initial network model meets the preset conditions. Then, the initial network model is determined to be trained and used as the target detection network model.

[0023] Secondly, embodiments of the present invention provide a target detection method based on the fusion of pixel and point cloud features, including:

[0024] Acquire point cloud and images of the driving environment of the target engineering vehicle;

[0025] The point cloud is projected onto the image to obtain a projected image;

[0026] The projected image is input into the target detection network model to detect multiple targets in the driving environment of the target engineering vehicle.

[0027] Thirdly, embodiments of the present invention provide a training apparatus for a target detection network model based on pixel and point cloud feature fusion, comprising:

[0028] The acquisition module is used to acquire point clouds and images of the driving environment of the engineering vehicle;

[0029] A projection module is used to project the point cloud onto the image to obtain a projected image;

[0030] The acquisition module is also used to acquire the pixel channel features of the projected image;

[0031] The training module is used to train the initial network model based on the pixel channel features to obtain the trained target detection network model.

[0032] Fourthly, embodiments of the present invention provide a target detection device based on pixel and point cloud feature fusion, comprising:

[0033] The acquisition module is used to acquire point cloud and images of the driving environment of the target engineering vehicle;

[0034] A projection module is used to project the point cloud onto the image to obtain a projected image;

[0035] The detection module is used to input the projected image into the target detection network model to detect multiple targets in the driving environment of the target engineering vehicle.

[0036] Fifthly, embodiments of the present invention provide an electronic device, including: a processor and a memory, wherein the processor is configured to execute a training program for a target detection network model based on pixel and point cloud feature fusion and a target detection program based on pixel and point cloud feature fusion stored in the memory, so as to implement the training method for the target detection network model based on pixel and point cloud feature fusion described in the first aspect and the target detection method based on pixel and point cloud feature fusion described in the second aspect.

[0037] In a sixth aspect, embodiments of the present invention provide a storage medium, comprising: the storage medium storing one or more programs, the one or more programs being executable by one or more processors to implement the training method for the target detection network model based on pixel and point cloud feature fusion described in the first aspect and the target detection method based on pixel and point cloud feature fusion described in the second aspect.

[0038] The training scheme for a target detection network model based on pixel and point cloud feature fusion provided in this invention involves acquiring point cloud and images of the driving environment of an engineering vehicle; projecting the point cloud onto the image to obtain a projected image; acquiring the pixel channel features of the projected image; and training an initial network model based on the pixel channel features to obtain a trained target detection network model. Compared to existing point cloud and image fusion technologies, which may introduce errors due to relative sensor motion, leading to inaccurate target detection results, this scheme obtains projected images by binding point clouds with different numbers of pixel channels. The target detection network model is then trained based on the projected images and their corresponding image features. This method avoids errors caused by relative sensor motion, resulting in more accurate target detection.

[0039] The target detection scheme based on pixel and point cloud feature fusion provided in this invention acquires the point cloud and image of the driving environment of the target engineering vehicle; projects the point cloud onto the image to obtain a projected image; and inputs the projected image into a target detection network model to detect multiple targets in the driving environment of the target engineering vehicle. This scheme, through the trained target detection network model, performs target detection in the driving environment of the target engineering vehicle, avoiding errors caused by relative motion of the sensors, accurately detecting targets and distinguishing target categories. Attached Figure Description

[0040] Figure 1 is a flowchart illustrating a training method for a target detection network model based on pixel and point cloud feature fusion according to an embodiment of the present invention.

[0041] Figure 2 is a flowchart illustrating another training method for a target detection network model based on pixel and point cloud feature fusion provided in an embodiment of the present invention.

[0042] Figure 3 is a schematic diagram of a pixel channel feature provided in an embodiment of the present invention;

[0043] Figure 4 is a schematic diagram of a feature module provided in an embodiment of the present invention;

[0044] Figure 5 is a schematic diagram of an FMN network experience mode provided by an embodiment of the present invention;

[0045] Figure 6 is a schematic diagram of an FMN network self-learning mode provided in an embodiment of the present invention;

[0046] Figure 7 is a flowchart illustrating a target detection method based on pixel and point cloud feature fusion provided in an embodiment of the present invention;

[0047] Figure 8 is a schematic diagram of the structure of a training device for a target detection network model based on pixel and point cloud feature fusion provided in an embodiment of the present invention;

[0048] Figure 9 is a schematic diagram of the structure of a target detection device based on pixel and point cloud feature fusion provided in an embodiment of the present invention;

[0049] Figure 10 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] To facilitate understanding of the embodiments of the present invention, further explanations and descriptions will be provided below with reference to the accompanying drawings and specific embodiments. These embodiments do not constitute a limitation on the embodiments of the present invention.

[0052] Figure 1 is a flowchart illustrating a training method for a target detection network model based on pixel and point cloud feature fusion according to an embodiment of the present invention. As shown in Figure 1, the method specifically includes:

[0053] S11. Obtain point cloud and image of the driving environment of the engineering vehicle.

[0054] In this embodiment of the invention, point clouds and images of the driving environment of the engineering vehicle can be acquired using a LiDAR sensor or an onboard camera. Point clouds and images of the driving environment in any direction in front of or around the engineering vehicle can be acquired.

[0055] It should be noted that point cloud and image data are corresponding data of the same target in the same time and space. The point cloud and image data can also be existing data stored in an open source database, or data collected on-site by engineering vehicles. This invention does not impose any specific limitations.

[0056] S12. Project the point cloud onto the image to obtain a projected image.

[0057] Color images are generally composed of three channels: R, G, and B. An image consisting of only one R, G, or B channel is grayscale, and a grayscale image is called a single-channel image. A pixel channel refers to the pixels of each channel. Under normal circumstances, point clouds can be perfectly projected onto the image. However, the movement of a vehicle causes relative motion between the sensor and the image, which generates projection errors. This results in misalignment of the point cloud projection onto the image, meaning that each point cloud cannot perfectly correspond to every pixel channel of the image.

[0058] To fully utilize the advantages of point clouds and images, this embodiment of the invention projects the point cloud onto the image. Specifically, the pixel channels of the point cloud and the image are aligned. When errors exist, alignment is performed according to preset rules to obtain the projected image.

[0059] S13. Obtain the pixel channel features of the projected image.

[0060] In this embodiment of the invention, an enhanced channel L can be selected to assist in the semantic recognition of the target category in the image. Image semantic segmentation is used to identify different target objects in the image, such as recognizing people as red and vehicles as blue. Each category corresponds to a color, resulting in pixel channel features. These pixel channel features characterize the target category and color features corresponding to each pixel channel.

[0061] S14. The initial network model is trained based on the pixel channel features to obtain the trained target detection network model.

[0062] The obtained pixel channel features are used to train the initial network model. The network construction mode can be divided into two types: empirical mode and self-learning mode. Figure 5 shows a schematic diagram of an empirical mode of the FMN network. The pixel channel features are fused and then learned by MLP, which can learn a large number of shallow features. These shallow features are then fused with the feature module in the FMN network. The feature module compensates for the loss of upper-layer semantic information as the network deepens, ensuring that image feature information is still preserved in deeper layers. Furthermore, the image pixel channels are split into R, G, and B modules, allowing the network to learn the data features of each channel separately, which also simplifies image feature learning. At the end of the FMN network, the R, G, and B modules are simultaneously refitted to further optimize and calibrate the learned data features, and the learned features are fused to obtain the optimal features. Figure 6 shows the self-learning mode. The FMN network is built using the NAS architecture search. The hyperparameter m represents the total number of modules. The order of each feature module and the number of each module are obtained by NAS self-learning, but the total number cannot exceed m. The image data features are learned through the self-learning mode.

[0063] Furthermore, the target object is determined from the features learned through any of the above methods, and the target detection result is output. Then, the accuracy of the target detection result can be used to determine whether the model training is complete. When the accuracy of the output result reaches the expected level, it can be indicated that the network model has completed its initial training. The model can then be tested using a pre-prepared test set. If the accuracy of the model's output on the test set also reaches the expected level, it indicates that the model training is complete, and the trained network model is used as the target detection network model.

[0064] It should be noted that because model training involves a large amount of computation and requires a large amount of raw data, the model training process can be completed on a cloud server. After the model is trained, it can be deployed on the autonomous driving domain controller or computing platform on the vehicle. The subsequent object detection process is completed on the domain controller or computing platform.

[0065] Optionally, after the model training is completed on the cloud server, the trained model is deployed on the cloud server. The vehicle-side domain controller obtains point clouds and images through sensors, performs preliminary processing, and then sends the processed data to the cloud server. The cloud server obtains the target detection results through the model and feeds them back to the vehicle-side domain controller.

[0066] The present invention provides a training method for a target detection network model based on pixel and point cloud feature fusion. This method involves acquiring point clouds and images of the driving environment of an engineering vehicle; projecting the point cloud onto the image to obtain a projected image; acquiring the pixel channel features of the projected image; and training an initial network model based on the pixel channel features to obtain a trained target detection network model. Compared to existing point cloud and image fusion techniques, which may introduce errors due to relative sensor motion, leading to inaccurate target detection results, this method controls the binding of the point cloud with different numbers of pixel channels to obtain a projected image. The target detection network model is then trained based on the projected image and its corresponding image features. This method avoids errors caused by relative sensor motion, resulting in more accurate target detection.

[0067] Figure 2 is a flowchart illustrating another training method for a target detection network model based on pixel and point cloud feature fusion provided in an embodiment of the present invention. As shown in Figure 2, the method specifically includes:

[0068] S21. When errors exist, the point cloud is bound to the pixel channels of the image based on preset rules to obtain a projected image.

[0069] In this embodiment of the invention, the hyperparameter q can be used to reduce the impact of errors. Specifically, when there are no errors, q is set to 1, and the point cloud is projected onto the image, meaning each point cloud is bound to a unique pixel channel in the image. When only horizontal or vertical errors exist, q can be set to 5, meaning each point cloud is bound to 5 pixel channels. When both horizontal and vertical errors exist, q can be set to 9, meaning each point cloud is bound to 9 pixel channels. The image after the point cloud and pixel channels are bound is used as the projected image.

[0070] S22. Use image semantic segmentation method to obtain the category of pixel channel bound to each point cloud.

[0071] In this embodiment of the invention, the enhanced channel L can be selected to assist in the semantic recognition of the target category of the image. Image semantic segmentation is used to identify different target objects in the image. For example, people in the image are identified as red, vehicles are identified as blue, etc. Each category corresponds to a color, thus obtaining the category of each pixel channel, and then obtaining the category of each pixel channel bound to the point cloud.

[0072] S23. Based on the category, determine the pixel channel features of the pixel region composed of pixel channels bound to each point cloud.

[0073] Furthermore, the pixel channel features of the pixel region composed of the pixel channels bound to each point cloud can be determined based on the category of the pixel channels bound to each point cloud. As shown in Figure 3, specifically, when there is no error, if a point cloud is bound to one pixel channel, the pixel channel features of that pixel channel can be determined; when there is an error, if a point cloud is bound to multiple pixel channels, and these multiple pixel channels form one or more pixel regions, the pixel channel features of the pixel region can be obtained by fusing the categories of the multiple pixel channels that make up the pixel region.

[0074] S24. Input the pixel channel features into a multilayer perceptron for feature mixing to obtain pixel channel mixed features.

[0075] Next, the pixel channel features are input into a multilayer perceptron for feature mixing, resulting in a feature module, as shown in Figure 4. This feature module fully incorporates the features learned by the network into the pixel channel features for mixing, resulting in pixel channel mixed features, which can provide some guidance for network fitting. In addition, semantic information that is only available in shallow layers is added to the deep network.

[0076] S25. Input the pixel channel hybrid features into the initial network model for model training until the output of the initial network model meets the preset conditions. Then, determine that the initial network model training is complete and use the trained initial network model as the target detection network model.

[0077] The obtained pixel channel fusion features are used to train the initial network model. The network construction mode can be divided into two types: empirical mode and self-learning mode. Figure 5 shows a schematic diagram of an empirical mode of the FMN network. The pixel channel features are fused and then learned by MLP, which can learn a large number of shallow features. These shallow features are then fused with the feature module in the FMN network. The feature module compensates for the loss of upper-layer semantic information as the network deepens, ensuring that image feature information is still preserved in deeper layers. Furthermore, the image pixel channels are split into R, G, and B modules, allowing the network to learn the data features of each channel separately, which also simplifies image feature learning. At the end of the FMN network, the R, G, and B modules are simultaneously refitted to further optimize and calibrate the learned data features, and the learned features are fused to obtain the optimal features. Figure 6 illustrates the self-learning mode. The FMN network is constructed using a NAS architecture search, where the hyperparameter m represents the total number of modules. The order and quantity of each feature module are obtained through NAS self-learning, but the total number cannot exceed m. The image data features are learned through this self-learning mode. Then, the target object is determined based on the learned features, and the target detection result is output. When the accuracy of the output result reaches the desired level, the network model training is considered complete, and the trained network model is used as the target detection network model.

[0078] The present invention provides a training method for a target detection network model based on pixel and point cloud feature fusion. This method acquires point clouds and images of the driving environment of an engineering vehicle; projects the point clouds onto the image to obtain a projected image; acquires the pixel channel features of the projected image; and trains an initial network model based on the pixel channel features to obtain a trained target detection network model. This method, by controlling the binding of point clouds with different numbers of pixel channels, avoids errors caused by relative sensor motion, resulting in more accurate target detection.

[0079] Figure 7 is a flowchart illustrating a target detection method based on pixel and point cloud feature fusion provided by an embodiment of the present invention. As shown in Figure 7, the method specifically includes:

[0080] S71. Obtain the point cloud and image of the driving environment of the target engineering vehicle.

[0081] In this embodiment of the invention, point clouds and images of the driving environment of the target engineering vehicle can be acquired using a LiDAR sensor or an onboard camera. Point clouds and images of the driving environment in any direction in front of or around the target engineering vehicle can be acquired.

[0082] S72. Project the point cloud onto the image to obtain a projected image.

[0083] Projecting the point cloud onto the image yields the projected image. Specifically, aligning the pixel channels of the point cloud with those of the image can be achieved by introducing a hyperparameter 'q'. In the absence of error, q equals 1, meaning each point cloud is bound to a unique pixel channel. When only lateral or vertical errors exist, q can equal 5, meaning each point cloud is bound to 5 pixel channels. When both lateral and vertical errors exist, q can equal 9, meaning each point cloud is bound to 9 pixel channels. The image after binding the point cloud with its pixel channels is used as the projected image.

[0084] S73. Input the projected image into the target detection network model to detect multiple targets in the driving environment of the target engineering vehicle.

[0085] The projected image obtained above is input into the trained target detection network model. The model analyzes the pixel channel features of the image and detects multiple targets in the driving environment of the target engineering vehicle, such as pedestrians and obstacles in front.

[0086] The target detection method based on pixel and point cloud feature fusion provided in this invention acquires point cloud and image of the driving environment of the target engineering vehicle; projects the point cloud onto the image to obtain a projected image; and inputs the projected image into a target detection network model to detect multiple targets in the driving environment of the target engineering vehicle. This method, through the trained target detection network model, performs target detection in the driving environment of the target engineering vehicle, avoiding errors caused by relative motion of the sensors, accurately detecting targets and distinguishing target categories.

[0087] Figure 8 is a schematic diagram of the structure of a training device for a target detection network model based on pixel and point cloud feature fusion provided in an embodiment of the present invention, specifically including:

[0088] The acquisition module 801 is used to acquire point cloud and images of the driving environment of the engineering vehicle. For detailed explanations, please refer to the relevant descriptions in the above method embodiments; they will not be repeated here.

[0089] The projection module 802 is used to project the point cloud onto the image to obtain a projected image. For detailed explanation, please refer to the relevant descriptions in the above method embodiments; they will not be repeated here.

[0090] The acquisition module 801 is further configured to acquire the pixel channel features of the projected image. For detailed explanations, please refer to the relevant descriptions in the above method embodiments; they will not be repeated here.

[0091] Training module 803 is used to train the initial network model based on the pixel channel features to obtain the trained object detection network model. For detailed explanations, please refer to the relevant descriptions in the above method embodiments; they will not be repeated here.

[0092] The training device for the target detection network model based on pixel and point cloud feature fusion provided in this embodiment can be the same as the training device for the target detection network model based on pixel and point cloud feature fusion shown in Figure 8. It can execute all the steps of the training method for the target detection network model based on pixel and point cloud feature fusion shown in Figure 1-2, thereby achieving the technical effect of the training method for the target detection network model based on pixel and point cloud feature fusion shown in Figure 1-2. For details, please refer to the relevant descriptions in Figure 1-2. For the sake of brevity, it will not be elaborated here.

[0093] Figure 9 is a schematic diagram of the structure of a target detection device based on pixel and point cloud feature fusion provided in an embodiment of the present invention, specifically including:

[0094] The acquisition module 901 is used to acquire point cloud and images of the driving environment of the target engineering vehicle. For detailed explanations, please refer to the relevant descriptions in the above method embodiments; they will not be repeated here.

[0095] The projection module 902 is used to project the point cloud onto the image to obtain a projected image. For detailed explanations, please refer to the relevant descriptions in the above method embodiments; they will not be repeated here.

[0096] The detection module 903 is used to input the projected image into the target detection network model to detect multiple targets in the driving environment of the target engineering vehicle. For detailed explanations, please refer to the relevant descriptions in the above method embodiments; they will not be repeated here.

[0097] The target detection device based on pixel and point cloud feature fusion provided in this embodiment can be the target detection device based on pixel and point cloud feature fusion as shown in Figure 9. It can execute all the steps of the target detection method based on pixel and point cloud feature fusion as shown in Figure 7, thereby achieving the technical effect of the target detection method based on pixel and point cloud feature fusion as shown in Figure 7. For details, please refer to the relevant description in Figure 7. For the sake of brevity, it will not be elaborated here.

[0098] Figure 10 is a schematic diagram of an electronic device according to an embodiment of the present invention. The electronic device 1000 shown in Figure 10 includes: at least one processor 1001, a memory 1002, at least one network interface 1004, and other user interfaces 1003. The various components in the electronic device 1000 are coupled together through a bus system 1005. It is understood that the bus system 1005 is used to realize the connection and communication between these components. In addition to a data bus, the bus system 1005 also includes a power bus, a control bus, and a status signal bus. However, for clarity, all buses are labeled as bus system 1005 in Figure 10.

[0099] The user interface 1003 may include a display, keyboard, or clicking device (e.g., mouse, trackball, touchpad, or touchscreen).

[0100] It is understood that the memory 1002 in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate Synchronous DRAM (DDRSDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The memory 1002 described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0101] In some implementations, memory 1002 stores elements, executable units or data structures, or subsets thereof, or extended sets thereof: operating system 10021 and application programs 10022.

[0102] The operating system 10021 includes various system programs, such as the framework layer, core library layer, and driver layer, used to implement various basic business functions and handle hardware-based tasks. The application program 10022 includes various applications, such as a media player and a browser, used to implement various application functions. The program implementing the method of this embodiment can be included in the application program 10022.

[0103] In this embodiment of the invention, by calling the program or instructions stored in the memory 1002, specifically the program or instructions stored in the application program 10022, the processor 1001 executes the method steps provided in each method embodiment, including, for example:

[0104] Acquire point cloud and image of the driving environment of the engineering vehicle; project the point cloud onto the image to obtain a projected image; acquire pixel channel features of the projected image; train an initial network model based on the pixel channel features to obtain a trained target detection network model.

[0105] In one possible implementation, when errors exist, the point cloud is bound to the pixel channels of the image based on preset rules to obtain a projected image.

[0106] In one possible implementation, when only lateral or longitudinal errors exist, each point cloud is bound to a first preset number of pixel channels; when both lateral and longitudinal errors exist, each point cloud is bound to a second preset number of pixel channels; and the image after binding the point cloud with the pixel channels is used as the projected image.

[0107] In one possible implementation, an image semantic segmentation method is used to obtain the category of the pixel channel bound to each point cloud; based on the category, the pixel channel features of the pixel region composed of the pixel channels bound to each point cloud are determined.

[0108] In one possible implementation, the pixel channel features are input into a multilayer perceptron for feature mixing to obtain pixel channel mixed features.

[0109] In one possible implementation, the pixel channel blending features are input into an initial network model for model training until the output of the initial network model meets a preset condition. Then, the initial network model is determined to be trained and the trained initial network model is used as the target detection network model.

[0110] or,

[0111] The point cloud and image of the driving environment of the target engineering vehicle are acquired; the point cloud is projected onto the image to obtain a projected image; the projected image is input into the target detection network model to detect multiple targets in the driving environment of the target engineering vehicle.

[0112] The methods disclosed in the above embodiments of the present invention can be applied to or implemented by processor 1001. Processor 1001 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 1001 or by instructions in the form of software. The processor 1001 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of the present invention can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software units in the decoding processor. The software units may be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 1002. Processor 1001 reads the information in memory 1002 and, in conjunction with its hardware, completes the steps of the above method.

[0113] It is understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described herein, or combinations thereof.

[0114] For software implementation, the techniques described herein can be implemented by units that perform the functions described herein. The software code can be stored in memory and executed by a processor. The memory can be implemented in the processor or external to the processor.

[0115] The electronic device provided in this embodiment can be the electronic device shown in Figure 10, which can execute all the steps of the target detection network model training method based on pixel and point cloud feature fusion in Figure 1-2 and the target detection method based on pixel and point cloud feature fusion in Figure 7, thereby achieving the technical effects of the target detection network model training method based on pixel and point cloud feature fusion shown in Figure 1-2 and the target detection method based on pixel and point cloud feature fusion shown in Figure 7. For details, please refer to the relevant descriptions in Figures 1-2 and 7. For the sake of brevity, they will not be elaborated here.

[0116] This invention also provides a storage medium (computer-readable storage medium). This storage medium stores one or more programs. The storage medium may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as read-only memory, flash memory, hard disk, or solid-state drive; the memory may also include combinations of the above types of memory.

[0117] When one or more programs in the storage medium can be executed by one or more processors to implement the above-mentioned training method for the target detection network model based on pixel and point cloud feature fusion and the target detection method based on pixel and point cloud feature fusion executed on the electronic device side.

[0118] The processor is used to execute the training program for the target detection network model based on pixel and point cloud feature fusion and the target detection program based on pixel and point cloud feature fusion stored in the memory, so as to realize the following steps of the training method for the target detection network model based on pixel and point cloud feature fusion and the target detection method based on pixel and point cloud feature fusion executed on the electronic device side:

[0119] Acquire point cloud and image of the driving environment of the engineering vehicle; project the point cloud onto the image to obtain a projected image; acquire pixel channel features of the projected image; train an initial network model based on the pixel channel features to obtain a trained target detection network model.

[0120] In one possible implementation, when errors exist, the point cloud is bound to the pixel channels of the image based on preset rules to obtain a projected image.

[0121] In one possible implementation, when only lateral or longitudinal errors exist, each point cloud is bound to a first preset number of pixel channels; when both lateral and longitudinal errors exist, each point cloud is bound to a second preset number of pixel channels; and the image after binding the point cloud with the pixel channels is used as the projected image.

[0122] In one possible implementation, an image semantic segmentation method is used to obtain the category of the pixel channel bound to each point cloud; based on the category, the pixel channel features of the pixel region composed of the pixel channels bound to each point cloud are determined.

[0123] In one possible implementation, the pixel channel features are input into a multilayer perceptron for feature mixing to obtain pixel channel mixed features.

[0124] In one possible implementation, the pixel channel blending features are input into an initial network model for model training until the output of the initial network model meets a preset condition. Then, the initial network model is determined to be trained and the trained initial network model is used as the target detection network model.

[0125] or,

[0126] The point cloud and image of the driving environment of the target engineering vehicle are acquired; the point cloud is projected onto the image to obtain a projected image; the projected image is input into the target detection network model to detect multiple targets in the driving environment of the target engineering vehicle.

[0127] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0128] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented in hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0129] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A training method for a target detection network model based on pixel and point cloud feature fusion, characterized in that, This includes: acquiring point cloud and images of the driving environment of the engineering vehicle; The point cloud is projected onto the image. When errors exist, the point cloud and the pixel channels of the image are bound together based on preset rules to obtain the projected image; the pixel channel features of the projected image are then obtained. The initial network model is trained based on the pixel channel features to obtain the trained target detection network model. When an error exists, the point cloud is bound to the pixel channels of the image based on a preset rule to obtain a projected image. This includes: when only a horizontal error or a vertical error exists, controlling each point cloud to bind a first preset number of pixel channels; when both the horizontal and vertical errors exist, controlling each point cloud to bind a second preset number of pixel channels; and using the image after binding the point cloud to the pixel channels as the projected image.

2. The method according to claim 1, characterized in that, The step of obtaining the pixel channel features of the projected image includes: using an image semantic segmentation method to obtain the category of the pixel channel bound to each point cloud; and based on the category, determining the pixel channel features of the pixel region composed of the pixel channels bound to each point cloud.

3. The method according to claim 2, characterized in that, The method further includes: inputting the pixel channel features into a multilayer perceptron for feature mixing to obtain pixel channel mixed features.

4. The method according to claim 3, characterized in that, The step of training the initial network model based on the pixel channel features to obtain the trained target detection network model includes: inputting the pixel channel mixed features into the initial network model for model training until the output of the initial network model meets the preset conditions, then determining that the initial network model training is complete, and using the trained initial network model as the target detection network model.

5. A target detection method based on pixel and point cloud feature fusion, characterized in that, include: Acquire point cloud and image of the driving environment of the target engineering vehicle; project the point cloud onto the image, and when only lateral error or longitudinal error exists, control each point cloud to bind a first preset number of pixel channels; When both the lateral and longitudinal errors exist simultaneously, control each point cloud to bind a second preset number of pixel channels; The image after binding the point cloud with the pixel channel is used as the projection image; the projection image is input into the target detection network model constructed as described in any one of claims 1-4 to detect multiple targets in the driving environment of the target engineering vehicle.

6. A training device for a target detection network model based on pixel and point cloud feature fusion, characterized in that, include: The acquisition module is used to acquire point clouds and images of the driving environment of the engineering vehicle; The projection module is used to project the point cloud onto the image. When only horizontal or vertical errors exist, it controls each point cloud to bind a first preset number of pixel channels. When both the lateral and longitudinal errors exist simultaneously, control each point cloud to bind a second preset number of pixel channels; The image after the point cloud and pixel channels are bound together is used as the projection image; the acquisition module is also used to acquire the pixel channel features of the projection image; The training module is used to train the initial network model based on the pixel channel features to obtain the trained target detection network model.

7. A target detection device based on pixel and point cloud feature fusion, characterized in that, include: The acquisition module is used to acquire point cloud and images of the driving environment of the target engineering vehicle; The projection module is used to project the point cloud onto the image. When only horizontal or vertical errors exist, it controls each point cloud to bind a first preset number of pixel channels. When both the lateral and longitudinal errors exist simultaneously, control each point cloud to bind a second preset number of pixel channels; The image after binding the point cloud with the pixel channel is used as the projection image; the detection module is used to input the projection image into the target detection network model to detect multiple targets in the driving environment of the target engineering vehicle.

8. An electronic device, characterized in that, include: A processor and a memory, the processor being configured to execute a training program for a target detection network model based on pixel and point cloud feature fusion and a target detection program based on pixel and point cloud feature fusion stored in the memory, to implement the training method for the target detection network model based on pixel and point cloud feature fusion as described in any one of claims 1 to 4 and the target detection method based on pixel and point cloud feature fusion as described in claim 5.

9. A storage medium, characterized in that, The storage medium stores one or more programs, which can be executed by one or more processors to implement the training method of the target detection network model based on pixel and point cloud feature fusion as described in any one of claims 1 to 4 and the target detection method based on pixel and point cloud feature fusion as described in claim 5.

Images