Object detection method, device and equipment

Abstract

The present application discloses an object detection method, device and equipment, and a vehicle driving information determination method, device and equipment. Among them, the object detection method includes: determining the first feature data of multiple first voxels corresponding to the first spatial resolution in the target space and the second feature data of multiple second voxels in the first voxel corresponding to at least one second spatial resolution respectively according to the environmental point cloud data, and using the first feature data and the second feature data as the third feature data of the first voxel; generating a two-dimensional feature image of the target space according to the third feature data through a two-dimensional feature extraction model corresponding to the network depth and the first spatial resolution; determining the object information in the target space according to the two-dimensional feature image including the adjacent information between the first voxels and the adjacent information between the second voxels. This processing method can effectively take into account both high object detection accuracy and fast detection speed.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and specifically to object detection methods, devices and equipment, vehicle driving information determination methods, devices and equipment, lesion detection methods, devices and equipment, virtual shopping methods, devices and equipment, and weather forecasting methods, devices and equipment. Background Art

[0002] In the fields of autonomous driving and robotics, machine perception is an important component, and perception sensors include laser radar, cameras, ultrasound, millimeter-wave radar, etc. Compared with sensors such as cameras, ultrasound, and millimeter-wave radar, the laser point cloud signal of multi-line laser radar contains accurate target position information and target geometry information, so it plays an important role in the perception of autonomous driving and robotics.

[0003] At present, a typical method of using laser point cloud to realize target recognition includes the following steps: 1) Different from the target detection algorithm that directly projects the laser point cloud to one or more planes, the 3D voxelization method divides the laser point cloud into multiple voxels of equal volume; 2) For each voxel, the voxel feature is extracted from the point cloud data of the voxel using methods such as MLP+pooling; 3) The spatial 3D information is reduced to spatial 2D information through multiple layers of 3D convolution; 4) Target detection is performed based on the reduced spatial 2D information. Compared with the current deep learning method based on laser point cloud projection (directly projecting the point cloud to a specific plane), this method improves the perception performance of the lidar because it fully extracts the adjacent information between voxels using 3D convolution.

[0004] In summary, the existing technology has the problem of not being able to take into account both high object detection accuracy and fast detection speed. How to improve object detection accuracy without reducing object detection speed has become an urgent problem that technicians in this field need to solve. Summary of the invention

[0005] The present application provides an object detection method to solve the problem that the prior art cannot achieve both high object detection accuracy and fast detection speed. The present application also provides an object detection device and equipment, a vehicle driving information determination method, device and equipment, a lesion detection method, device and equipment, a virtual shopping method, device and equipment, and a weather prediction method, device and equipment.

[0006] The present application provides an object detection method, comprising:

[0007] Determine, according to the environmental point cloud data, first feature data of a plurality of first voxels corresponding to a first spatial resolution in the target space and second feature data of a plurality of second voxels in the first voxels corresponding to at least one second spatial resolution respectively, and use the first feature data and the second feature data as third feature data of the first voxel;

[0008] Using the third feature data of the first voxel as input data of a two-dimensional feature extraction model corresponding to the network depth and the first spatial resolution, and generating a two-dimensional feature image of the target space through the two-dimensional feature extraction model;

[0009] According to the two-dimensional feature image including the first voxel-to-voxel neighboring information and the second voxel-to-voxel neighboring information, object information in the target space is determined.

[0010] Optionally, the first characteristic data and the second characteristic data are determined by the following steps:

[0011] Dividing the target space into a plurality of first voxels according to the environmental point cloud data;

[0012] dividing the first voxel into a plurality of second voxels;

[0013] The first feature data is determined according to the point cloud data of the first voxel; and the second feature data is determined according to the point cloud data of the second voxel.

[0014] Optionally, also include:

[0015] determining a region of interest within the target space;

[0016] Clear the point cloud data outside the area of ​​interest in the environmental point cloud data.

[0017] Optionally, generating a two-dimensional feature image of the target space by using the two-dimensional feature extraction model includes:

[0018] Split the three-dimensional convolution kernel of the image feature extraction layer based on manifold sparse convolution into multiple single convolution kernels;

[0019] For the first single convolution kernel arranged before the central single convolution kernel in the image feature extraction layer, if there is an output non-empty voxel j generated by the first single convolution kernel k for the input non-empty voxel i, then a convolution index relationship R (i, k, j) between the input non-empty voxel, the first single convolution kernel and the output non-empty voxel is constructed;

[0020] For the second single convolution kernel arranged after the central single convolution kernel in the image feature extraction layer, if there is a corresponding relationship R(i, k, j), a convolution index relationship R(j, k, i) between the input non-empty voxel j, the second single convolution kernel k and the output non-empty voxel i is constructed;

[0021] Construct the convolution index relationship R(i, k, i) between the input non-empty voxel i, the central single convolution kernel k and the output non-empty voxel i;

[0022] According to the convolution index relationship, the single convolution kernel and the input non-empty voxels, the non-empty voxels output by the manifold sparse convolution-based image feature extraction layer are determined.

[0023] Optionally, generating a two-dimensional feature image of the target space by using the two-dimensional feature extraction model includes:

[0024] Split the three-dimensional convolution kernel of the image feature extraction layer based on non-manifold sparse convolution into multiple single convolution kernels;

[0025] Determine the global index j of the output voxel based on the spatial size, padding and convolution step of the convolution kernel;

[0026] For an input non-empty voxel i, if there is an output non-empty voxel j generated by a single convolution kernel k, then a convolution index relationship R(i, k, i) between the input non-empty voxel i, the single convolution kernel k and the output non-empty voxel j is constructed;

[0027] According to the convolution index relationship, the single convolution kernel and the input non-empty voxels, the non-empty voxels output by the image feature extraction layer based on non-manifold sparse convolution are determined.

[0028] Optionally, the objects include: vehicles, people, and obstacles.

[0029] The present application provides an object detection device, comprising:

[0030] a feature data determination unit, configured to determine, based on the environmental point cloud data, first feature data of a plurality of first voxels corresponding to a first spatial resolution in a target space and second feature data of a plurality of second voxels in the first voxels corresponding to at least one second spatial resolution, respectively, and use the first feature data and the second feature data as third feature data of the first voxels;

[0031] a two-dimensional feature extraction unit, configured to use the third feature data of the first voxel as input data of a two-dimensional feature extraction model corresponding to a network depth and a first spatial resolution, and generate a two-dimensional feature image of a target space through the two-dimensional feature extraction model;

[0032] The object information determining unit is used to determine the object information in the target space according to the two-dimensional feature image including the first voxel-to-voxel neighboring information and the second voxel-to-voxel neighboring information.

[0033] Optionally, also include:

[0034] An area of ​​interest determining unit, used to determine an area of ​​interest in the target space;

[0035] The data clearing unit is used to clear the point cloud data outside the area of ​​interest in the environmental point cloud data.

[0036] The present application provides a device, comprising:

[0037] Processor; and

[0038] A memory is used to store a program for implementing an object detection method. After the device is powered on and the program of the method is run through the processor, the following steps are performed: first feature data of multiple first voxels corresponding to a first spatial resolution in a target space and second feature data of multiple second voxels in the first voxel corresponding to at least one second spatial resolution are determined based on environmental point cloud data, and the first feature data and the second feature data are used as third feature data of the first voxel; the third feature data of the first voxel is used as input data of a two-dimensional feature extraction model corresponding to a network depth and the first spatial resolution, and a two-dimensional feature image of the target space is generated through the two-dimensional feature extraction model; object information in the target space is determined based on the two-dimensional feature image including adjacent information between first voxels and adjacent information between second voxels.

[0039] Optionally, the equipment includes a vehicle; the vehicle includes a three-dimensional space scanning device.

[0040] The present application provides a method for determining vehicle driving information, comprising:

[0041] Determine, according to the environmental point cloud data, first feature data of a plurality of first voxels corresponding to a first spatial resolution in the target space and second feature data of a plurality of second voxels in the first voxels corresponding to at least one second spatial resolution respectively, and use the first feature data and the second feature data as third feature data of the first voxel;

[0042] Using the third feature data of the first voxel as input data of a two-dimensional feature extraction model corresponding to the network depth and the first spatial resolution, and generating a two-dimensional feature image of the target space through the two-dimensional feature extraction model;

[0043] Determine object information in the target space according to the two-dimensional feature image including the first voxel adjacent information and the second voxel adjacent information; the object information includes the current position information of the first vehicle;

[0044] The driving speed and / or driving track of the first vehicle is determined according to the current position information and the historical position information of the first vehicle.

[0045] Optionally, also include:

[0046] The driving mode of the second vehicle is adjusted according to the driving speed and / or the driving trajectory.

[0047] The present application provides an object detection method, comprising:

[0048] Determine, according to the environmental point cloud data, first feature data of a plurality of first voxels corresponding to a first spatial resolution in the target space and second feature data of a plurality of second voxels in the first voxels corresponding to at least one second spatial resolution respectively, and use the first feature data and the second feature data as third feature data of the first voxel;

[0049] Using the third feature data of the first voxel as input data of a two-dimensional feature extraction model corresponding to the network depth and the first spatial resolution, and generating a two-dimensional feature image of the target space through the two-dimensional feature extraction model;

[0050] Determine first object information in the target space according to the two-dimensional feature image including first voxel-to-voxel neighboring information and second voxel-to-voxel neighboring information;

[0051] Second object information in the target space is determined based on the environmental perception data outside the environmental point cloud data and the first object information.

[0052] Optionally, the perception data includes at least one of the following data: a two-dimensional image, and millimeter wave radar perception data.

[0053] The present application provides a vehicle driving information determination device, comprising:

[0054] a feature data determination unit, configured to determine, based on the environmental point cloud data, first feature data of a plurality of first voxels corresponding to a first spatial resolution in a target space and second feature data of a plurality of second voxels in the first voxels corresponding to at least one second spatial resolution, respectively, and use the first feature data and the second feature data as third feature data of the first voxels;

[0055] a two-dimensional feature extraction unit, configured to use the third feature data of the first voxel as input data of a two-dimensional feature extraction model corresponding to a network depth and a first spatial resolution, and generate a two-dimensional feature image of a target space through the two-dimensional feature extraction model;

[0056] A position information determining unit, configured to determine object information in the target space according to the two-dimensional feature image including the first voxel-to-voxel adjacent information and the second voxel-to-voxel adjacent information; the object information includes the current position information of the first vehicle;

[0057] The driving information determination unit is used to determine the driving speed and / or driving trajectory of the first vehicle based on the current position information and historical position information of the first vehicle.

[0058] The present application provides an object detection device, comprising:

[0059] a feature data determination unit, configured to determine, based on the environmental point cloud data, first feature data of a plurality of first voxels corresponding to a first spatial resolution in a target space and second feature data of a plurality of second voxels in the first voxels corresponding to at least one second spatial resolution, respectively, and use the first feature data and the second feature data as third feature data of the first voxels;

[0060] a two-dimensional feature extraction unit, configured to use the third feature data of the first voxel as input data of a two-dimensional feature extraction model corresponding to a network depth and a first spatial resolution, and generate a two-dimensional feature image of a target space through the two-dimensional feature extraction model;

[0061] A first object information determining unit, configured to determine first object information in a target space according to a two-dimensional feature image including first voxel-to-voxel neighboring information and second voxel-to-voxel neighboring information;

[0062] The second object information determining unit is used to determine the second object information in the target space according to the environmental perception data outside the environmental point cloud data and the first object information.

[0063] The present application provides a device, comprising:

[0064] Processor; and

[0065] A memory is used to store a program for implementing a method for determining vehicle driving information. After the device is powered on and runs the program of the method through the processor, the following steps are performed: based on the environmental point cloud data, first feature data of multiple first voxels corresponding to the first spatial resolution in the target space and second feature data of multiple second voxels in the first voxel corresponding to at least one second spatial resolution respectively are determined, and the first feature data and the second feature data are used as third feature data of the first voxel; the third feature data of the first voxel is used as input data of a two-dimensional feature extraction model corresponding to the network depth and the first spatial resolution, and a two-dimensional feature image of the target space is generated through the two-dimensional feature extraction model; object information in the target space is determined based on the two-dimensional feature image including adjacent information between first voxels and adjacent information between second voxels; the object information includes current position information of the first vehicle; and the driving speed and / or driving trajectory of the first vehicle is determined based on the current position information and historical position information of the first vehicle.

[0066] The present application provides a device, comprising:

[0067] Processor; and

[0068] A memory is used to store a program for implementing an object detection method. After the device is powered on and the program of the method is run through the processor, the following steps are performed: first feature data of multiple first voxels corresponding to a first spatial resolution in a target space and second feature data of multiple second voxels in the first voxel corresponding to at least one second spatial resolution are determined according to environmental point cloud data, and the first feature data and the second feature data are used as third feature data of the first voxel; the third feature data of the first voxel is used as input data of a two-dimensional feature extraction model corresponding to a network depth and the first spatial resolution, and a two-dimensional feature image of the target space is generated through the two-dimensional feature extraction model; first object information in the target space is determined according to the two-dimensional feature image including adjacent information between first voxels and adjacent information between second voxels; second object information in the target space is determined according to environmental perception data outside the environmental point cloud data and the first object information.

[0069] The present application also provides an object detection method, comprising:

[0070] Acquire a two-dimensional environment image of the target space;

[0071] Constructing a three-dimensional environment image of the target space according to the two-dimensional environment image;

[0072] Determine, according to the three-dimensional environment image, first feature data of a plurality of first voxels corresponding to a first spatial resolution in a target space and second feature data of a plurality of second voxels in the first voxels corresponding to at least one second spatial resolution respectively, and use the first feature data and the second feature data as third feature data of the first voxel;

[0073] Using the third feature data of the first voxel as input data of a two-dimensional feature extraction model corresponding to the network depth and the first spatial resolution, and generating a two-dimensional feature image of the target space through the two-dimensional feature extraction model;

[0074] According to the two-dimensional feature image including the first voxel-to-voxel neighboring information and the second voxel-to-voxel neighboring information, object information in the target space is determined.

[0075] The present application also provides a lesion detection method, comprising:

[0076] acquiring at least one computerized tomography (CT) image of the target site;

[0077] constructing a three-dimensional image of the target site based on at least one CT image;

[0078] Determine, according to the three-dimensional image, first feature data of a plurality of first voxels corresponding to a first spatial resolution in the target part and second feature data of a plurality of second voxels in the first voxels corresponding to at least one second spatial resolution respectively, and use the first feature data and the second feature data as third feature data of the first voxel;

[0079] Using the third feature data of the first voxel as input data of a two-dimensional feature extraction model corresponding to the network depth and the first spatial resolution, and generating a two-dimensional feature image of the target part through the two-dimensional feature extraction model;

[0080] The lesion information of the target part is determined according to the two-dimensional feature image including the first voxel-to-voxel adjacent information and the second voxel-to-voxel adjacent information.

[0081] The present application also provides an object detection device, comprising:

[0082] A two-dimensional image acquisition unit, used to acquire a two-dimensional environment image of the target space;

[0083] A three-dimensional image construction unit, used to construct a three-dimensional environment image of the target space according to the two-dimensional environment image;

[0084] a feature data determining unit, configured to determine, based on the three-dimensional environment image, first feature data of a plurality of first voxels corresponding to a first spatial resolution in a target space and second feature data of a plurality of second voxels in the first voxels corresponding to at least one second spatial resolution, respectively, and use the first feature data and the second feature data as third feature data of the first voxels;

[0085] a two-dimensional feature extraction unit, configured to use the third feature data of the first voxel as input data of a two-dimensional feature extraction model corresponding to a network depth and a first spatial resolution, and generate a two-dimensional feature image of a target space through the two-dimensional feature extraction model;

[0086] The object information determining unit is used to determine the object information in the target space according to the two-dimensional feature image including the first voxel-to-voxel neighboring information and the second voxel-to-voxel neighboring information.

[0087] The present application also provides a device, comprising:

[0088] Processor; and

[0089] A memory is used to store a program for implementing an object detection method. After the device is powered on and the program of the method is run through the processor, the following steps are performed: acquiring a two-dimensional environment image of the target space; constructing a three-dimensional environment image of the target space based on the two-dimensional environment image; determining first feature data of multiple first voxels corresponding to a first spatial resolution in the target space and second feature data of multiple second voxels in the first voxel corresponding to at least one second spatial resolution respectively based on the three-dimensional environment image, and using the first feature data and the second feature data as third feature data of the first voxel; using the third feature data of the first voxel as input data of a two-dimensional feature extraction model corresponding to the network depth and the first spatial resolution, and generating a two-dimensional feature image of the target space through the two-dimensional feature extraction model; determining object information in the target space based on the two-dimensional feature image including adjacent information between the first voxels and adjacent information between the second voxels.

[0090] The present application also provides a lesion detection device, comprising:

[0091] An image acquisition unit, used to acquire at least one computerized tomography (CT) image of a target site;

[0092] A three-dimensional image construction unit, configured to construct a three-dimensional image of a target part according to at least one CT image;

[0093] a feature data determining unit, configured to determine, based on the three-dimensional image, first feature data of a plurality of first voxels corresponding to a first spatial resolution in the target part and second feature data of a plurality of second voxels in the first voxels corresponding to at least one second spatial resolution, respectively, and use the first feature data and the second feature data as third feature data of the first voxels;

[0094] a two-dimensional feature extraction unit, configured to use the third feature data of the first voxel as input data of a two-dimensional feature extraction model corresponding to a network depth and a first spatial resolution, and generate a two-dimensional feature image of a target part through the two-dimensional feature extraction model;

[0095] The lesion information determining unit is used to determine the lesion information of the target part according to the two-dimensional feature image including the first voxel adjacent information and the second voxel adjacent information.

[0096] The present application also provides a device, comprising:

[0097] Processor; and

[0098] A memory is used to store a program for implementing a lesion detection method. After the device is powered on and the program of the method is run through the processor, the following steps are performed: acquiring at least one electronic computer tomography (CT) image of the target part; constructing a three-dimensional image of the target part based on the at least one CT image; determining, based on the three-dimensional image, first feature data of multiple first voxels corresponding to a first spatial resolution in the target part and second feature data of multiple second voxels in the first voxel corresponding to at least one second spatial resolution respectively, and using the first feature data and the second feature data as third feature data of the first voxel; using the third feature data of the first voxel as input data of a two-dimensional feature extraction model corresponding to a network depth and a first spatial resolution, and generating a two-dimensional feature image of the target part through the two-dimensional feature extraction model; determining lesion information of the target part based on the two-dimensional feature image including adjacent information between first voxels and adjacent information between second voxels.

[0099] The present application also provides a virtual shopping method, comprising:

[0100] Determine, according to the shopping environment point cloud data, first feature data of a plurality of first voxels corresponding to a first spatial resolution in the shopping space and second feature data of a plurality of second voxels in the first voxels corresponding to at least one second spatial resolution respectively, and use the first feature data and the second feature data as third feature data of the first voxels;

[0101] Using the third feature data of the first voxel as input data of a two-dimensional feature extraction model corresponding to the network depth and the first spatial resolution, and generating a two-dimensional feature image of the shopping space through the two-dimensional feature extraction model;

[0102] determining object information in the shopping space according to the two-dimensional feature image including the first voxel-to-voxel adjacent information and the second voxel-to-voxel adjacent information;

[0103] According to the object information, a virtual object is placed into the three-dimensional image of the shopping space.

[0104] The present application also provides a weather prediction method, comprising:

[0105] Collect three-dimensional spatial images of target clouds;

[0106] Determine, according to the three-dimensional spatial image, first feature data of a plurality of first voxels corresponding to a first spatial resolution in the target cloud layer and second feature data of a plurality of second voxels in the first voxels corresponding to at least one second spatial resolution respectively, and use the first feature data and the second feature data as third feature data of the first voxel;

[0107] Using the third feature data of the first voxel as input data of a two-dimensional feature extraction model corresponding to a network depth and a first spatial resolution, and generating a two-dimensional feature image of a target cloud layer through the two-dimensional feature extraction model;

[0108] Weather information is determined based on a two-dimensional feature image including first inter-voxel neighboring information and second inter-voxel neighboring information.

[0109] The present application also provides a virtual shopping device, comprising:

[0110] a feature data determining unit, configured to determine, based on the shopping environment point cloud data, first feature data of a plurality of first voxels corresponding to a first spatial resolution in the shopping space and second feature data of a plurality of second voxels in the first voxels corresponding to at least one second spatial resolution respectively, and use the first feature data and the second feature data as third feature data of the first voxels;

[0111] a two-dimensional feature extraction unit, configured to use the third feature data of the first voxel as input data of a two-dimensional feature extraction model corresponding to a network depth and a first spatial resolution, and generate a two-dimensional feature image of the shopping space through the two-dimensional feature extraction model;

[0112] An object information determination unit, configured to determine object information in the shopping space based on a two-dimensional feature image including first voxel-to-voxel neighboring information and second voxel-to-voxel neighboring information;

[0113] The virtual object adding unit is used to place the virtual object into the three-dimensional image of the shopping space according to the object information.

[0114] The present application also provides a device, comprising:

[0115] Processor; and

[0116] A memory is used to store a program for implementing a virtual shopping method. After the device is powered on and the program of the method is run through the processor, the following steps are performed: first feature data of multiple first voxels corresponding to a first spatial resolution in a shopping space and second feature data of multiple second voxels in the first voxel corresponding to at least one second spatial resolution are determined based on shopping environment point cloud data, and the first feature data and the second feature data are used as third feature data of the first voxel; the third feature data of the first voxel is used as input data of a two-dimensional feature extraction model corresponding to a network depth and the first spatial resolution, and a two-dimensional feature image of the shopping space is generated through the two-dimensional feature extraction model; object information in the shopping space is determined based on the two-dimensional feature image including adjacent information between first voxels and adjacent information between second voxels; and virtual objects are placed into the three-dimensional image of the shopping space based on the object information.

[0117] The present application also provides a weather prediction device, comprising:

[0118] An image acquisition unit, used for acquiring a three-dimensional spatial image of a target cloud layer;

[0119] a feature data determining unit, configured to determine, based on the three-dimensional spatial image, first feature data of a plurality of first voxels corresponding to a first spatial resolution in a target cloud layer and second feature data of a plurality of second voxels in the first voxels corresponding to at least one second spatial resolution, respectively, and use the first feature data and the second feature data as third feature data of the first voxels;

[0120] a two-dimensional feature extraction unit, configured to use the third feature data of the first voxel as input data of a two-dimensional feature extraction model corresponding to a network depth and a first spatial resolution, and generate a two-dimensional feature image of a target cloud layer through the two-dimensional feature extraction model;

[0121] The weather information determination unit is used to determine the weather information according to the two-dimensional feature image including the first voxel-to-voxel adjacent information and the second voxel-to-voxel adjacent information.

[0122] The present application also provides a device, comprising:

[0123] Processor; and

[0124] A memory is used to store a program for implementing a weather prediction method. After the device is powered on and the program of the method is run through the processor, the following steps are performed: acquiring a three-dimensional spatial image of a target cloud layer; determining, based on the three-dimensional spatial image, first feature data of a plurality of first voxels corresponding to a first spatial resolution in the target cloud layer and second feature data of a plurality of second voxels in the first voxel corresponding to at least one second spatial resolution, and using the first feature data and the second feature data as third feature data of the first voxel; using the third feature data of the first voxel as input data of a two-dimensional feature extraction model corresponding to a network depth and a first spatial resolution, and generating a two-dimensional feature image of the target cloud layer through the two-dimensional feature extraction model; and determining weather information based on the two-dimensional feature image including adjacent information between first voxels and adjacent information between second voxels.

[0125] The present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, and when the computer-readable storage medium is run on a computer, the computer executes the above-mentioned various methods.

[0126] The present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the above-mentioned various methods.

[0127] Compared with the prior art, this application has the following advantages:

[0128] The object detection method provided in the embodiment of the present application determines, according to the environmental point cloud data, first feature data of multiple first voxels corresponding to the first spatial resolution in the target space and second feature data of multiple second voxels in the first voxel corresponding to at least one second spatial resolution, and uses the first feature data and the second feature data as the third feature data of the first voxel; uses the third feature data of the first voxel as input data of a two-dimensional feature extraction model corresponding to the network depth and the first spatial resolution, and generates a two-dimensional feature image of the target space through the two-dimensional feature extraction model; determines the object information in the target space according to the two-dimensional feature image including the adjacent information between the first voxels and the adjacent information between the second voxels; this processing method enables three-dimensional image dimensionality reduction processing based on coarse-grained voxels to generate a two-dimensional feature image, and divides the coarse-grained voxels more finely, so that the feature data of the coarse-grained voxel includes not only the voxel feature data of the coarse-grained voxel itself, but also the voxel feature data of the fine-grained voxels therein, so that more sufficient original information of the point cloud is retained in the feature data of the coarse-grained voxel, thereby avoiding the loss of part of the object information; therefore, it is possible to effectively take into account both high object detection accuracy and fast detection speed. In addition, this multi-voxel partitioning method occupies less system resources.

[0129] The vehicle driving information determination method provided in the embodiment of the present application determines, based on the environmental point cloud data, first feature data of multiple first voxels corresponding to the first spatial resolution in the target space and second feature data of multiple second voxels in the first voxel corresponding to at least one second spatial resolution respectively, and uses the first feature data and the second feature data as the third feature data of the first voxel; uses the third feature data of the first voxel as input data of a two-dimensional feature extraction model corresponding to the network depth and the first spatial resolution, and generates a two-dimensional feature image of the target space through the two-dimensional feature extraction model; determines object information in the target space based on the two-dimensional feature image including adjacent information between the first voxels and the second voxels; the object information includes the first and second spatial resolutions; The present invention relates to a method for determining the current position information of a vehicle; determining the driving speed and / or driving trajectory of the first vehicle according to the current position information and historical position information of the first vehicle; this processing method enables dimensionality reduction processing of three-dimensional images based on coarse-grained voxels to generate a two-dimensional feature image, and more finely divides the coarse-grained voxels, so that the feature data of the coarse-grained voxels not only includes the voxel feature data of the coarse-grained voxels themselves, but also includes the voxel feature data of the fine-grained voxels therein, so that more sufficient original information of the point cloud is retained in the feature data of the coarse-grained voxels, thereby avoiding the loss of part of the vehicle information, and determining the driving speed and / or driving trajectory of the vehicle according to more accurate vehicle information; therefore, it is possible to effectively take into account both a higher accuracy of vehicle driving information and a faster speed of determining vehicle driving information.

[0130] The object detection method provided by the embodiment of the present application determines, based on environmental point cloud data, first feature data of multiple first voxels corresponding to a first spatial resolution in a target space and second feature data of multiple second voxels in the first voxel corresponding to at least one second spatial resolution, and uses the first feature data and the second feature data as third feature data of the first voxel; uses the third feature data of the first voxel as input data of a two-dimensional feature extraction model corresponding to a network depth and the first spatial resolution, and generates a two-dimensional feature image of the target space through the two-dimensional feature extraction model; determines first object information in the target space based on the two-dimensional feature image including adjacent information between the first voxels and adjacent information between the second voxels; and determines the first object information in the target space based on the two-dimensional feature image outside the environmental point cloud data. The environmental perception data and the first object information are used to determine the second object information in the target space; this processing method enables three-dimensional image dimensionality reduction processing based on coarse-grained voxels to generate a two-dimensional feature image, and the coarse-grained voxels are divided more finely, so that the feature data of the coarse-grained voxels not only include the voxel feature data of the coarse-grained voxels themselves, but also include the voxel feature data of the fine-grained voxels therein, so that more sufficient original point cloud information is retained in the feature data of the coarse-grained voxels, thereby avoiding the loss of some object information, and based on the first object information, further according to other perception data other than the point cloud (such as two-dimensional images, millimeter-wave radar perception data, etc.) to determine more accurate second object information; therefore, it is possible to effectively take into account both higher object detection accuracy and faster detection speed.

[0131] The object detection method provided by the embodiment of the present application obtains a two-dimensional environment image of the target space; constructs a three-dimensional environment image of the target space according to the two-dimensional environment image; determines first feature data of multiple first voxels corresponding to a first spatial resolution in the target space and second feature data of multiple second voxels in the first voxel corresponding to at least one second spatial resolution respectively according to the three-dimensional environment image, and uses the first feature data and the second feature data as third feature data of the first voxel; uses the third feature data of the first voxel as input data of a two-dimensional feature extraction model corresponding to the network depth and the first spatial resolution, and extracts the first feature data through the two-dimensional feature extraction. The model generates a two-dimensional feature image of the target space; according to the two-dimensional feature image including the adjacent information between the first voxels and the adjacent information between the second voxels, the object information in the target space is determined; this processing method enables the three-dimensional image to be reduced in dimension based on the coarse-grained voxels to generate a two-dimensional feature image, and the coarse-grained voxels are divided more finely, so that the feature data of the coarse-grained voxels not only includes the voxel feature data of the coarse-grained voxels themselves, but also includes the voxel feature data of the fine-grained voxels therein, so that more sufficient spatial information is retained in the feature data of the coarse-grained voxels, thereby avoiding the loss of some object information; therefore, it is possible to effectively take into account both high object detection accuracy and fast detection speed. In addition, by adopting this processing method, there is no need to collect point cloud data through a laser radar, but the object information can be directly determined based on the two-dimensional image collected by the camera, so the equipment cost can be effectively reduced. In addition, compared with the technical solution of the prior art that directly determines object information based on the collected two-dimensional image through an object detection model (such as the RefineDet method based on deep learning, etc.), the processing method of the present invention is adopted, so that the input data (two-dimensional feature image) of the object detection model includes spatial information, and the object information determined according to the two-dimensional feature image is more accurate.

[0132] The lesion detection method provided in the embodiment of the present application obtains at least one electronic computer tomography (CT) image of the target part; constructs a three-dimensional image of the target part based on the at least one CT image; determines, based on the three-dimensional image, first feature data of a plurality of first voxels corresponding to a first spatial resolution in the target part and second feature data of a plurality of second voxels in the first voxels corresponding to at least one second spatial resolution, respectively, and uses the first feature data and the second feature data as third feature data of the first voxel; uses the third feature data of the first voxel as input data of a two-dimensional feature extraction model corresponding to the network depth and the first spatial resolution, and extracts the lesion through the two-dimensional feature extraction. The model generates a two-dimensional feature image of the target part; according to the two-dimensional feature image including the adjacent information between the first voxels and the adjacent information between the second voxels, the lesion information of the target part is determined; this processing method performs image dimensionality reduction processing on the three-dimensional image of the target part based on the coarse-grained voxels to generate a two-dimensional feature image, and divides the coarse-grained voxels more finely, so that the feature data of the coarse-grained voxels not only includes the voxel feature data of the coarse-grained voxels themselves, but also includes the voxel feature data of the fine-grained voxels therein, so that more sufficient spatial information is retained in the feature data of the coarse-grained voxels, thereby avoiding the loss of some lesion information; therefore, it is possible to effectively take into account both higher lesion detection accuracy and faster detection speed.

[0133] The virtual shopping method provided by the embodiment of the present application determines, based on the shopping environment point cloud data, first feature data of multiple first voxels corresponding to the first spatial resolution in the shopping space and second feature data of multiple second voxels in the first voxel corresponding to at least one second spatial resolution respectively, and uses the first feature data and the second feature data as the third feature data of the first voxel; uses the third feature data of the first voxel as input data of a two-dimensional feature extraction model corresponding to the network depth and the first spatial resolution, and generates a two-dimensional feature image of the shopping space through the two-dimensional feature extraction model; and generates a two-dimensional feature image of the shopping space according to the two-dimensional feature map including the adjacent information between the first voxels and the adjacent information between the second voxels. image, determine the object information in the shopping space; according to the object information, place the virtual object into the three-dimensional image of the shopping space; this processing method performs image dimensionality reduction processing on the three-dimensional image of the target shopping space based on the coarse-grained voxels to generate a two-dimensional feature image, and divides the coarse-grained voxels more finely, so that the feature data of the coarse-grained voxels not only includes the voxel feature data of the coarse-grained voxels themselves, but also includes the voxel feature data of the fine-grained voxels therein, so that more sufficient spatial information is retained in the feature data of the coarse-grained voxels, thereby avoiding the loss of some product information; therefore, it is possible to effectively take into account both higher object detection accuracy and faster detection speed, thereby improving product sales rate and user experience.

[0134] The weather prediction method provided in the embodiment of the present application collects a three-dimensional spatial image of a target cloud layer; determines, based on the three-dimensional spatial image, first feature data of a plurality of first voxels corresponding to a first spatial resolution in the target cloud layer and second feature data of a plurality of second voxels in the first voxel corresponding to at least one second spatial resolution, and uses the first feature data and the second feature data as third feature data of the first voxel; uses the third feature data of the first voxel as input data of a two-dimensional feature extraction model corresponding to a network depth and the first spatial resolution, and generates a two-dimensional feature map of the target cloud layer through the two-dimensional feature extraction model. image; determining weather information according to a two-dimensional feature image including adjacent information between the first voxels and adjacent information between the second voxels; this processing method performs image dimensionality reduction processing on the three-dimensional image of the target cloud layer based on the coarse-grained voxels to generate a two-dimensional feature image, and divides the coarse-grained voxels more finely, so that the feature data of the coarse-grained voxels not only includes the voxel feature data of the coarse-grained voxels themselves, but also includes the voxel feature data of the fine-grained voxels therein, so that more sufficient spatial information is retained in the feature data of the coarse-grained voxels, thereby avoiding the loss of some weather information; therefore, it is possible to effectively take into account both higher weather forecast accuracy and faster detection speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0135] Figure 1 is a flow chart of an embodiment of the object detection method provided by the present application;

[0136] Figure 2 is a schematic diagram of voxel division of an embodiment of the object detection method provided by the present application;

[0137] Figure 3 is a schematic diagram of a network structure of an embodiment of an object detection method provided by the present application;

[0138] Figure 4 is a specific flow chart of an embodiment of the object detection method provided by the present application;

[0139] Figure 5 is a schematic diagram of an embodiment of an object detection device provided by the present application;

[0140] Figure 6 is a schematic diagram of an embodiment of the device provided by the present application;

[0141] Figure 7 is a flow chart of an embodiment of a method for determining vehicle driving information provided by the present application;

[0142] Figure 8 is a flow chart of an embodiment of the object detection method provided by the present application;

[0143] Fig. 9is a flow chart of an embodiment of the object detection method provided by the present application;

[0144] Fig.10 is a flow chart of an embodiment of a lesion detection method provided by the present application;

[0145] Fig.11 is a flow chart of an embodiment of a virtual shopping method provided by the present application;

[0146] Fig.12 It is a flow chart of an embodiment of the weather prediction method provided by the present application. DETAILED DESCRIPTION

[0147] Many specific details are described in the following description to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present application, so the present application is not limited by the specific implementation disclosed below.

[0148] In the present application, object detection methods, devices and equipment, vehicle driving information determination methods, devices and equipment, lesion detection methods, devices and equipment, virtual shopping methods, devices and equipment, weather prediction methods, devices and equipment are provided. Each scheme is described in detail in the following embodiments.

[0149] First embodiment

[0150] Please refer to Figure 1 , which is a flow chart of an embodiment of an object detection method provided by the present application. The execution subject of the method includes but is not limited to unmanned vehicles, such as intelligent logistics vehicles, etc., and the identifiable objects include traffic objects, such as pedestrians, vehicles, obstacles, etc. on the road, and other objects can also be identified. The following takes the detection of traffic objects as an example to illustrate the method. An object detection method provided by the present application includes:

[0151] Step S101: Determine, based on the environmental point cloud data, first feature data of multiple first voxels corresponding to a first spatial resolution in the target space and second feature data of multiple second voxels in the first voxels corresponding to at least one second spatial resolution, and use the first feature data and the second feature data as the third feature data of the first voxel.

[0152] The method provided in the embodiment of the present application can obtain the spatial coordinates of each sampling point on the surface of the environmental space object of the road where the vehicle is traveling through a three-dimensional space scanning device installed on the vehicle during the driving process of the vehicle (hereinafter referred to as the self-vehicle), and obtain a set of points. This massive point data is called road environment point cloud (Point Cloud) data. Through the road environment point cloud data, the scanned object surface is recorded in the form of points, and each point contains three-dimensional coordinates, some of which may contain color information (RGB) or reflection intensity information (Intensity). With point cloud data, the target space can be expressed in the same spatial reference system.

[0153] The three-dimensional space scanning device may be a laser radar (Light Detection And Ranging, Lidar), which performs laser detection and measurement by laser scanning to obtain information about obstacles in the surrounding environment, such as buildings, trees, people, vehicles, etc. The measured data is a discrete point representation of a digital surface model (DSM). In specific implementation, a multi-line laser radar such as 16-line, 32-line, or 64-line can be used. The frame rate (Frame Rate) of point cloud data collected by radars with different numbers of laser beams is different. For example, 16 and 32 lines generally collect 10 frames of point cloud data per second. The three-dimensional space scanning device may also be a three-dimensional laser scanner or a photographic scanner.

[0154] The road environment point cloud data of this embodiment includes the spatial coordinate data of the sampling points of the surface points of the objects in the laser radar sensing area. Based on the road environment point cloud data, the point cloud data of multiple first voxels corresponding to the first spatial resolution in the sensing area (target space) can be determined. Voxel is the abbreviation of volume pixel, which can be used to represent a unit of display basic points in three-dimensional space, similar to a pixel in a two-dimensional plane. The volume of the first voxel depends on the first spatial resolution and can be a coarser-grained first volume.

[0155] It should be noted that in the process of implementing the present invention, the inventors found that the existing technical solutions have at least the following problems: on the one hand, the accuracy of object detection is directly related to the volume of voxels. The finer the voxelization, the smaller the volume of the voxel, and the more fully the voxel can retain the original information of the point cloud, thus having a higher object detection accuracy; on the other hand, the running speed of the 3D convolutional network is also directly related to the size of the input data after voxelization. The finer the voxelization, the larger the amount of input data, thereby increasing the video memory burden of the image processing unit (GPU) and reducing the running speed of the 3D convolutional network; on the other hand, the network depth of the 3D convolutional network is also directly related to the volume of the voxel. The smaller the voxel, the smaller the field of view covered by each voxel. In this case, in order to achieve a sufficiently large receptive field required to detect large targets, the network needs a greater depth, resulting in a decrease in network efficiency.

[0156] The difference between the method provided in this embodiment and the prior art includes: the prior art only performs one-level voxel division on the perception area, while the method provided in this embodiment can perform multi-level voxel division on the perception area. For example, the method provided in this embodiment can first divide the perception area into multiple first voxels of the first spatial resolution, and the multiple first voxels usually have the same volume, and the volume of the first voxel is called the first volume; then, each first voxel is further voxel-divided. In this embodiment, the subvoxel of the first voxel is called the second voxel, and the spatial resolution of the second voxel is called the second spatial resolution, which is usually higher than the first spatial resolution. The second voxels corresponding to at least one second spatial resolution usually have different volumes, and the volume of the second voxel is called the second volume.

[0157] In one example, the first voxel may be divided into a plurality of second voxels of different second volumes, that is, the first voxel may be divided according to a plurality of second spatial resolutions. For example, first, the laser point cloud is divided into first voxels of relatively large granularity, such as the volume of each first voxel is 0.1*0.1*0.2 meters; then, for each coarse-grained first voxel, it may be further divided into fine-grained second voxels, such as Figure 2 As shown, it can be divided into 8 voxels of 2*2*2, and / or 27 voxels of 3*3*3, and higher order voxels of n*n*n. 3 For each coarse-grained first voxel, its description feature (i.e., the third feature data) is F = [f 1,1 ,f 8,1 f 8,2 ,…,f 8,8 ,f 27,1 ,f 27,2 ,…,f 27,27 ,…], where f i,jIt is the feature of the jth subvoxel when the voxel is divided into i parts. Voxel features can use some simple features, such as the average value of the points inside the voxel Or complex features, such as features generated by MLP+pooling.

[0158] In another example, the first voxel may be divided into a plurality of second voxels of the same second volume, that is, the first voxel is divided at only one second spatial resolution. For example, the spatial resolution of the coarser-grained first voxel is 0.1*0.1*0.2 meters, and the spatial resolution of the second voxel divided into smaller ones is 0.025*0.025*0.05 meters, and the second voxel is 1 / 4 of the first voxel in a single dimension.

[0159] In specific implementation, step S101 may include the following sub-steps: 1) dividing the target space (perception area) into multiple first voxels based on the environmental point cloud data; 2) dividing the first voxels into multiple second voxels; 3) determining the first feature data based on the point cloud data of the first voxels; and determining the second feature data based on the point cloud data of the second voxels.

[0160] In a specific implementation, the first feature data of each first voxel and the second feature data of each second voxel can be determined through a voxel feature learning network, and the first feature data and the second feature data are used as the third feature data of the first voxel. Since the voxel feature learning network is a relatively mature existing technology, it will not be described here.

[0161] In one example, step S101 can be implemented in the following manner: if the multi-level voxel division condition is met, the first feature data and the second feature data are determined based on the environmental point cloud data, and the first feature data and the second feature data are used as the third feature data of the first voxel; if the multi-level voxel division condition is not met, the first feature data is determined based on the environmental point cloud data, and the first feature data is used as input data of the two-dimensional feature extraction model, and the object information in the target space is determined based on the two-dimensional feature image including adjacent information between the first voxels.

[0162] The multi-level voxel division conditions include but are not limited to at least one of the following conditions: the object recognition accuracy is greater than the accuracy threshold; the object information includes vehicles, people or obstacles on the road; the resource occupancy rate of the processor executing the method is greater than the occupancy rate threshold.

[0163] The accuracy threshold can be determined according to business requirements. For example, if the accuracy threshold is 95%, and if the object recognition accuracy is required to be 98%, the hierarchical voxel division condition is met, and multi-level voxel division is required.

[0164] For another example, in a business scenario where an unmanned vehicle automatically identifies obstacles on the road, since autonomous driving requires higher safety and needs to avoid frequent braking caused by misidentification, it has a higher object recognition accuracy requirement. This scenario meets the conditions for multi-level voxel division.

[0165] This processing method allows the multi-level voxel division working mode to be switched in real time according to actual conditions; therefore, computing resources and storage resources can be effectively saved, thereby reducing energy consumption and hardware costs.

[0166] For another example, if the processor resources (CPU or GPU occupancy rate) of the current vehicle system are high, in order to avoid a crash caused by an overly busy CPU or GPU, if this situation is not improved, multi-level voxel division should not be performed. This can sacrifice the accuracy of some object recognition in exchange for the normal operation of the processor. When the processor is not busy, it can switch to the multi-level voxel division working mode in real time. In this embodiment, the method is executed in the GPU, occupying only a small number of system resources.

[0167] In another example, the second feature data can be determined by the following steps: 1) dividing the target space into a first region that requires multi-voxel division and a second region that does not require multi-voxel division; 2) determining second feature data of multiple second voxels in the first voxel within the first region that respectively correspond to at least one second spatial resolution.

[0168] For example, the first area is defined as the area between the road surface and a spatial position where the spatial height is less than a height threshold. The height threshold may be 4 meters, so that various objects on the road can be basically identified, such as double-decker buses and the like.

[0169] This processing method ensures that object information can be identified while only performing multi-level voxel division on part of the area; therefore, it can effectively save computing resources and storage resources, thereby reducing energy consumption and hardware costs.

[0170] In another example, the third feature data is determined by the following steps: 1) determining the area within the first voxel to be divided into the second voxel, for example, the area of ​​interest for autonomous driving determined based on the high-precision map and the real-time position of the autonomous driving vehicle, including roads, sidewalks, drivable areas, etc.; 2) using the first feature data of the first voxel and the second feature data within the area to be divided into the second voxel as the third feature data.

[0171] In practical applications, some of the first voxels include some areas that do not need to be divided into second voxels. For example, areas that will not affect autonomous driving are determined based on high-precision maps and the real-time position of autonomous driving vehicles, such as roadside fences and areas outside the fences. In this case, there is no need to divide this area into second voxels, nor is there a need for second feature data for this area.

[0172] By adopting this processing method, after a first voxel is divided into multiple levels, only part of the second feature data is taken as the third feature data, so as to ensure the sparsity of the input data input to the two-dimensional feature extraction model as much as possible; therefore, the object recognition speed can be effectively improved.

[0173] Step S103: using the third feature data of the first voxel as input data of a two-dimensional feature extraction model corresponding to the network depth and the first spatial resolution, and generating a two-dimensional feature image of the target space through the two-dimensional feature extraction model.

[0174] The input data of the two-dimensional feature extraction model includes three-dimensional feature data, and the output data is a two-dimensional feature image. The two-dimensional feature image includes spatial information of the target space, and this information is arranged in a two-dimensional matrix in a two-dimensional manner.

[0175] The differences between the method provided in this embodiment and the prior art also include: 1) if the same object detection accuracy is to be achieved, the network depth of the two-dimensional feature extraction model in the prior art must be higher than the network depth of the two-dimensional feature extraction model in the method provided in this embodiment; 2) the original point cloud information retained in the two-dimensional feature image generated by the two-dimensional feature extraction model in the prior art is far less than the original point cloud information retained in the two-dimensional feature image generated by the two-dimensional feature extraction model in the method provided in this embodiment.

[0176] 1) If the same object detection accuracy is to be achieved, the network depth of the two-dimensional feature extraction model in the prior art must be higher than the network depth of the two-dimensional feature extraction model in the method provided in this embodiment.

[0177] In the prior art, to achieve higher object detection accuracy, it is necessary to use voxels with smaller granularity, higher spatial resolution, and larger number for 3D convolution, which will result in a deeper 3D convolution network depth. For example, the spatial resolution of coarser-grained voxels is 0.1*0.1*0.2 meters. If the spatial resolution of smaller voxels is 0.025*0.025*0.05 meters, the smaller voxels are 1 / 4 of the larger voxels in a single dimension. To achieve the same receptive field, it is necessary to expand the receptive field at the resolution four times. Among them, the increased network depth varies depending on the network structure. For example, if the pooling layer is used to increase the receptive field, it is necessary to add two convolution blocks with pooling, where the convolution block includes multiple convolution layers and pooling layers.

[0178] like Figure 3 As shown, in the method provided in this embodiment, the first voxel of the third feature data with relatively large granularity, relatively low spatial resolution but sufficient original information of the point cloud is used for 3D convolution, rather than the second voxel with relatively small granularity being used for 3D convolution. In this way, the total number of voxels input into the two-dimensional feature extraction model can be controlled. Under the premise of ensuring high object detection accuracy, the network depth of the two-dimensional feature extraction model can be effectively reduced, thereby improving the object detection speed. In other words, both high object detection accuracy and faster detection speed can be taken into account.

[0179] 2) The original point cloud information retained in the two-dimensional feature image generated by the two-dimensional feature extraction model in the method provided in this embodiment is much higher than the original point cloud information retained in the two-dimensional feature image in the prior art.

[0180] The method provided in this embodiment performs object recognition based on a two-dimensional feature image with more sufficient original point cloud information, so the detected object information is more accurate. In the prior art, a two-dimensional feature image only includes adjacent information between voxels of one volume; in this embodiment, the two-dimensional feature image includes not only adjacent information between the first voxels, but also adjacent information between the second voxels of at least one volume. It can be seen that the two-dimensional feature image of this embodiment has sufficient original point cloud information, and the object information recognized based on the two-dimensional feature image with sufficient original point cloud information is more accurate.

[0181] Please see Figure 4 , which is a specific flow chart of an embodiment of the object detection method of the present application. In this embodiment, the method may also include the following steps:

[0182] Step S401: determining a region of interest in the target space.

[0183] The region of interest (ROI) includes but is not limited to areas such as vehicle roads and sidewalks that have an impact on vehicle driving; correspondingly, non-interested areas may include areas occupied by objects such as buildings and trees on both sides of the road.

[0184] Step S403: clearing point cloud data outside the area of ​​interest in the environmental point cloud data.

[0185] The region of interest can be determined in the following manner: first, the location information of the device is determined by high-precision positioning of the point cloud data acquisition device; then, the ROI in the target space is determined according to the location information through the high-precision map ROI determination module; finally, the point cloud data outside the ROI is filtered out through the ROI filtering module.

[0186] The method provided in this embodiment filters out point cloud data outside the ROI from the environmental point cloud data, and executes steps S101 to S105 according to the remaining ROI point cloud data. This processing method greatly reduces the number of points that need to be processed, and can filter out approximately 50% or more of irrelevant points. Therefore, it can effectively save computing resources and storage resources, thereby improving the object detection speed.

[0187] It should be noted that, compared with the 2D convolution operation in the target detection process in step S105, the 3D convolution operation in the dimensionality reduction process of the three-dimensional image in step S103 has a larger amount of calculation. For the effective perception distance range of [-64, 64] meters in the x (front and back) direction, [-64, 64] meters in the y (left and right) direction, [-3, 1] meters in the z (up and down) direction, and the first spatial resolution of [0.1, 0, 1, 0.2] meters, the spatial dimension of the input feature of the generated two-dimensional feature extraction model is 1280*1280*20 dimensions. Such high-dimensional input data is a great burden for 2D convolution, and it will incur a great time cost for 3D convolution. Compared with the 2D image convolution algorithm, LiDAR has strong sparsity. For a 32-line LiDAR, a frame of laser point cloud contains 57,600 laser points, and the corresponding non-empty first voxels are about 25,000. In this case, about 99.92% of the first voxels are empty. If the laser point cloud outside the ROI is filtered out through a high-precision map, the proportion of empty first voxels will be higher. In this case, sparse 3D convolution can effectively improve the convolution speed, thereby improving the object detection speed.

[0188] In summary, real-time performance has become a very important issue for 3D convolution, which can be alleviated by sparse 3D convolution. However, the real-time performance of existing algorithms still needs to be further improved. How to achieve faster sparse convolution to further improve the speed of object detection has become an urgent problem that technicians in this field need to solve.

[0189] There are two types of sparse 3D convolutions: one is ordinary sparse convolution, which has no specific restrictions on the output of convolution; the other is manifold sparse convolution, which generates output at the corresponding output position only when there are activated voxels (non-empty voxels) at the corresponding input position. Among them, ordinary 3D convolution will have more and more activated voxels as the number of convolution layers increases, so the sparsity of the data will deteriorate, thus affecting the efficiency of 3D convolution calculation. Manifold sparse convolution can ensure that the output layer and the input layer have the same number of activated voxels, so that no additional activated voxels will be generated during the sparse convolution process, thereby ensuring the performance of the calculation. In sparse network design, ordinary sparse convolution layers are usually combined with manifold sparse convolution layers, such as combining one ordinary sparse convolution layer with three manifold sparse convolution layers to form a convolution block, which ensures the extraction of features between adjacent voxels and suppresses the rapid growth of the number of activated voxels, thereby ensuring network performance (i.e. object detection accuracy) and calculation speed.

[0190] The process of sparse convolution is as follows. First, the non-empty voxel regions are grouped together by indexing them. Then, the general matrix multiplication (GEMM) algorithm is used to implement fast convolution. Finally, the convolution output is returned to the corresponding 3D output position. However, due to the high sparsity of LiDAR data, the voxels x in the same convolution kernel are P(j),l It is still possible that there are a large number of empty voxels. For example, if the convolution kernel is a 3D convolution of 3*3*3, there may be more than 50% empty voxels among the corresponding 3*3*3=27 input voxels, which is still a huge computational burden.

[0191] To solve this problem, step S103 may include the following sub-steps:

[0192] Step S1031: split the three-dimensional convolution kernel of the image feature extraction layer based on manifold sparse convolution into multiple single convolution kernels.

[0193] The method provided in this embodiment further expands the convolution kernel according to the spatial dimension. For example, a convolution kernel with a spatial dimension of 3*3*3 can be split into 27 single convolution kernels, a convolution kernel with a spatial dimension of 5*5*5 can be split into 125 single convolution kernels, and so on. In this case, the corresponding sparse convolution GEMM mathematical expression is:

[0194]

[0195] Where k is the index value of a single convolution kernel, the number of index values ​​is the number of single convolution kernels, w is the weight of a single convolution kernel, l is the feature dimension of the input data, and m is the feature dimension of the output data. R(i,k,j) represents the convolution index relationship of the input voxel i through the kth single convolution kernel to produce the output j, x R(i,k,j),l It is the aggregation of the input layer data corresponding to the output at the j-th position.

[0196] Step S1033: For the first single convolution kernel arranged before the central single convolution kernel in the image feature extraction layer, if there is an output non-empty voxel j generated by the first single convolution kernel k for the input non-empty voxel i, then a convolution index relationship R (i, k, j) between the input non-empty voxel, the first single convolution kernel and the output non-empty voxel is constructed.

[0197] Step S1035: For the second single convolution kernel arranged after the central single convolution kernel in the image feature extraction layer, if a convolution index relationship R(i, k, j) exists, a convolution index relationship R(j, k, i) is constructed between the input non-empty voxel j, the second single convolution kernel k and the output non-empty voxel i.

[0198] Sparse convolution requires a step to establish a convolution index, which can be achieved by traversal. The traversal process is: for a sparse convolution with a convolution kernel of 3*3*3, for each non-empty voxel, traverse its 3*3*3 neighborhood, find other non-empty voxels in the neighborhood, and establish a corresponding relationship. For example, for i non-empty voxels, there is a j non-empty voxel at the k-th position in the 3*3*3 neighborhood, then establish an index such as (i, k, j). Traditional methods require searching every position in the neighborhood to establish an index relationship.

[0199] The core part of the method provided in this embodiment to implement fast sparse convolution is to quickly establish the index relationship R between input i, output j and single convolution kernel k. Taking the spatial size of the convolution kernel as 3*3*3 as an example, for manifold sparse convolution, the inventors found two laws in the process of implementing the present invention:

[0200] Law 1: If there exists a corresponding relationship (i, k, j) between input i, single convolution kernel k, and output j, that is, if input i generates output j through single convolution kernel k, then there must exist a relationship (j, 26-k, i) where input j generates output i through single convolution kernel 26-k.

[0201] Law 2: For the central convolution kernel, such as the 3*3*3 convolution kernel center is a single convolution kernel 13, the corresponding index relationship is only {(i,13,i),i∈[0,N)}.

[0202] Through Law 1 and Law 2, we only need to search and establish the corresponding relationship (i, k, j) under a single convolution kernel k∈[0,13) to establish all the corresponding relationships of a single convolution kernel k∈[0,27), so that the time required to establish the index can be reduced by half.

[0203] For example, assuming that the number of voxels in the input layer is N, and the spatial size of the convolution kernel of an output channel is 3*3*3=27, the process of establishing the manifold sparse convolution index relationship R may include the following steps:

[0204] Step 1: Establish the same global index for the output voxels as the input layer, and set the index of the empty voxel to -1;

[0205] For example, the input layer includes 10 non-empty voxels, whose global positions are 50, 200, 1000, 9400, 90700, 109000, 180000, 370000, 500000, 700000, and the corresponding global indices are 0 to 9; correspondingly, the global voxel positions of the output voxels are 50, 200, 1000, 9400, 90700, 109000, 180000, 370000, 500000, 700000, and the global indices are also 0 to 9 accordingly.

[0206] Step 2: For each input non-empty voxel i, determine whether the output j generated by the convolution kernel k∈[0,13) exists. If so, store (i, k, j) in the index relationship R.

[0207] For example, the judgment method is: taking the 3*3*3 convolution kernel as an example, for the non-empty voxel i, judge the 13 neighbors in the first half of the 27 neighbors of i in the space, and the relative positions are (0, 0, 0), (0, 0, 1), (0, 0, 2), (0, 1, 0), (0, 1, 1), (0, 1, 2), (0, 2, 0), (0, 2, 1), (0, 2, 2), (1, 0, 0), (1, 0, 1), (1, 0, 2), (1, 1, 0), 13 neighboring voxels, numbered k from 0 to 12. If there is a non-empty voxel j in the kth neighboring position, establish the index (i, k, j).

[0208] Step 3: Use Law 1 and Law 2 to complete the index relation R of k∈[13,27).

[0209] First, according to Law 2, the index (i, 13, j) of the center point can be added; then, if (i, k, j) exists at the kth neighbor position k∈[0,13), then according to Law 1, (i, 26-k, j) exists, and this relationship is added to the index relationship R.

[0210] The method provided in the embodiment of the present application, through the symmetry law discovered by the inventor, only needs to search half of the neighborhood space to complete the index establishment.

[0211] Step S1037: Determine the non-empty voxels output by the manifold sparse convolution-based image feature extraction layer according to the convolution index relationship, the single convolution kernel and the input non-empty voxels.

[0212] After quickly constructing the index relationship in the above manner, a general matrix multiplication algorithm can be used to implement fast convolution based on the convolution index relationship, the single convolution kernel and the non-empty voxels of the input, determine the non-empty voxels of the output, and return the convolution output to the corresponding 3D output position.

[0213] The above describes a method for quickly establishing a streaming sparse convolution index relationship R. Next, a method for quickly establishing a common sparse convolution index relationship R is described.

[0214] For ordinary sparse convolution, there is no need to restrict the position of the output voxel, so the consistency of the input and output voxel positions does not exist. In this case, Law 1 and Law 2 are not suitable for ordinary sparse convolution. To quickly establish the ordinary sparse convolution index relationship R, step S103 may include the following sub-steps:

[0215] Step S1031 ′: splitting the three-dimensional convolution kernel of the image feature extraction layer based on non-manifold sparse convolution into multiple single convolution kernels.

[0216] Step S1033 ′: Determine the global index j of the output voxel according to the spatial size, padding and convolution step of the single convolution kernel.

[0217] For example, for a common convolution with a convolution kernel size of 3*3*3, the same input and output size, padding of 0, and a convolution step of 1, for the input non-empty voxel i, due to the diffusion of the convolution kernel, the 27 neighboring spatial positions of 3*3*3 centered on i are all the outputs of the corresponding output layer. The output voxel positions corresponding to all input non-empty voxels are determined, and these output voxels are assigned a unique global index j.

[0218] In order to assign a unique global index j to the output layer, first establish a query table of the same size as the output layer, set the values ​​of all voxels in the table to -1, and if an output voxel is an activated output and is assigned a global index, set the voxel query table value to -2. By judging whether the value in the query table is -1 or -2, ensure that the activated voxel is assigned a unique global index.

[0219] Step S1035': for each input voxel i, determine whether the output j generated by the single convolution kernel k exists, and if so, store (i, k, j) in the index relationship R.

[0220] Taking the 3*3*3 convolution kernel as an example, for the i-th input voxel, determine the output voxel j in the output layer neighborhood corresponding to the voxel, and establish the index relationship (i, k, j). For example, if the convolution kernel size is 3*3*3, the input and output are the same size, the padding is 0, and the convolution step is 1, for the input non-empty voxel i, the 27 voxels at the corresponding position of the output layer need to be indexed.

[0221] Step S1037': Determine the non-empty voxels output by the image feature extraction layer based on non-manifold sparse convolution according to the convolution index relationship, the single convolution kernel and the input non-empty voxels.

[0222] Step S105: determining object information in the target space according to the two-dimensional feature image including the first voxel-to-voxel neighboring information and the second voxel-to-voxel neighboring information.

[0223] The method provided by this embodiment is different from the prior art in that the two-dimensional feature image in the prior art only includes adjacent information between voxels of one volume, while the two-dimensional feature image in the method provided by this embodiment retains both adjacent information between the first voxels and adjacent information between the second voxels. The object information in the perception area is determined based on the image, which can effectively improve the accuracy of higher object detection.

[0224] In specific implementation, the object information in the target space can be determined by an image-based target detection algorithm (such as Faster R-CNN, SSD, FCN, etc.) according to a two-dimensional feature image including the first voxel-to-voxel adjacent information and the second voxel-to-voxel adjacent information. The object in this embodiment can be a traffic object. The traffic object can be a vehicle, a person, or an obstacle such as a tree.

[0225] In this embodiment, the object information in the target space is determined according to the two-dimensional feature image through the traffic object detection model. The traffic object detection model can adopt the RefineDet method based on deep learning. This method combines the two-stage method such as Faster R-CNN with the fast running speed of the single-stage method such as SSD, so it has the advantage of high object detection accuracy. When the method detects the image of the traffic object (the traffic object is an obstacle for the moving vehicle) in the traffic environment image, the bounding box coordinates of the traffic object image are obtained, that is, the position data of the traffic object image in the traffic environment image. The position data can be the vertex coordinate data of the 3D rectangular bounding box of the traffic object image, that is, the position data can be a 24-dimensional vector, which respectively represents the x coordinate, y coordinate, and z coordinate of all 8 vertices. After determining the traffic object image in the two-dimensional feature image, the object detection result can also be converted from the vehicle coordinate system to the world coordinate system, so as to determine the spatial object information. Since the object detection technology for the two-dimensional feature image belongs to the relatively mature existing technology, it will not be repeated here.

[0226] It can be seen from the above embodiments that the object detection method provided in the embodiments of the present application determines, based on the environmental point cloud data, first feature data of multiple first voxels corresponding to the first spatial resolution in the target space and second feature data of multiple second voxels in the first voxel corresponding to at least one second spatial resolution respectively, and uses the first feature data and the second feature data as the third feature data of the first voxel; uses the third feature data of the first voxel as input data of a two-dimensional feature extraction model corresponding to the network depth and the first spatial resolution, and generates a two-dimensional feature image of the target space through the two-dimensional feature extraction model; based on According to the two-dimensional feature image including the adjacent information between the first voxels and the adjacent information between the second voxels, the object information in the target space is determined; this processing method enables the three-dimensional image to be reduced in dimension based on the coarse-grained voxels to generate a two-dimensional feature image, and the coarse-grained voxels are divided more finely, so that the feature data of the coarse-grained voxels not only includes the voxel feature data of the coarse-grained voxels themselves, but also includes the voxel feature data of the fine-grained voxels therein, so that more sufficient original information of the point cloud is retained in the feature data of the coarse-grained voxels, thereby avoiding the loss of some object information; therefore, it is possible to effectively take into account both higher object detection accuracy and faster detection speed.

[0227] Second embodiment

[0228] In the above embodiment, an object recognition method is provided, and correspondingly, the present application also provides an object detection device, which corresponds to the embodiment of the above method.

[0229] Please see Figure 5 , which is a schematic diagram of an embodiment of an object detection device of the present application. Since the device embodiment is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described below is only illustrative.

[0230] The present application further provides an object detection device, comprising:

[0231] The feature data determining unit 501 is used to determine, based on the environmental point cloud data, first feature data of a plurality of first voxels corresponding to a first spatial resolution in the target space and second feature data of a plurality of second voxels in the first voxels corresponding to at least one second spatial resolution respectively, and use the first feature data and the second feature data as third feature data of the first voxels;

[0232] A two-dimensional feature extraction unit 503 is used to use the third feature data of the first voxel as input data of a two-dimensional feature extraction model corresponding to the network depth and the first spatial resolution, and generate a two-dimensional feature image of the target space through the two-dimensional feature extraction model;

[0233] The object information determining unit 505 is used to determine the object information in the target space according to the two-dimensional feature image including the first voxel-to-voxel neighboring information and the second voxel-to-voxel neighboring information.

[0234] Optionally, the objects include: vehicles, people, and obstacles.

[0235] Optionally, also include:

[0236] An area of ​​interest determining unit, used to determine an area of ​​interest in the target space;

[0237] The data clearing unit is used to clear the point cloud data outside the area of ​​interest in the environmental point cloud data.

[0238] Optionally, the feature data determining unit includes:

[0239] A first voxel division subunit is used to divide the target space into a plurality of first voxels according to the environmental point cloud data;

[0240] A second voxel division subunit, configured to divide the first voxel into a plurality of second voxels;

[0241] A first feature data determining subunit, configured to determine the first feature data according to the point cloud data of the first voxel;

[0242] The second feature data determination subunit is used to determine the second feature data according to the point cloud data of the second voxel.

[0243] Optionally, the two-dimensional feature extraction unit includes:

[0244] A convolution kernel splitting subunit is used to split the three-dimensional convolution kernel of the image feature extraction layer based on manifold sparse convolution into multiple single convolution kernels;

[0245] A first index relationship construction subunit is used for constructing a convolution index relationship R (i, k, j) between the input non-empty voxel, the first single convolution kernel and the output non-empty voxel if there is an output non-empty voxel j generated by the first single convolution kernel k for the input non-empty voxel i.

[0246] A second index relationship construction subunit is used for constructing a convolution index relationship R(j, k, i) between the input non-empty voxel j, the second single convolution kernel k and the output non-empty voxel i for the second single convolution kernel arranged after the central single convolution kernel in the image feature extraction layer, if there is a corresponding relationship R(i, k, j);

[0247] The third index relationship construction subunit is used to construct a convolution index relationship R (i, k, i) between the input non-empty voxel i, the central single convolution kernel k and the output non-empty voxel i;

[0248] The convolution operation subunit is used to determine the non-empty voxels output by the image feature extraction layer based on manifold sparse convolution according to the convolution index relationship, the single convolution kernel and the input non-empty voxels.

[0249] Optionally, the two-dimensional feature extraction unit includes:

[0250] A convolution kernel splitting subunit, used for splitting the three-dimensional convolution kernel of the image feature extraction layer based on non-manifold sparse convolution into multiple single convolution kernels;

[0251] A global index determination subunit, used to determine the global index j of the output voxel according to the spatial size, padding and convolution step of the convolution kernel;

[0252] An index relationship construction subunit is used to construct a convolution index relationship R(i, k, i) between the input non-empty voxel i, the single convolution kernel k and the output non-empty voxel j if there is an output non-empty voxel j generated by a single convolution kernel k.

[0253] The convolution operation subunit is used to determine the non-empty voxels output by the image feature extraction layer based on non-manifold sparse convolution according to the convolution index relationship, the single convolution kernel and the input non-empty voxels.

[0254] Third embodiment

[0255] In the above embodiment, an object detection method is provided, and correspondingly, the present application also provides a device. The embodiment of the device corresponds to the embodiment of the above method.

[0256] Please see Figure 6 , which is a schematic diagram of an embodiment of the device of the present application. Since the device embodiment is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described below is only illustrative.

[0257] The present application further provides a device, including: a processor 601; and a memory 602, for storing a program for implementing an object detection method. After the device is powered on and runs the program of the method through the processor, the following steps are performed: based on the environmental point cloud data, first feature data of multiple first voxels corresponding to the first spatial resolution in the target space and second feature data of multiple second voxels in the first voxel corresponding to at least one second spatial resolution respectively, are determined, and the first feature data and the second feature data are used as third feature data of the first voxel; the third feature data of the first voxel is used as input data of a two-dimensional feature extraction model corresponding to the network depth and the first spatial resolution, and a two-dimensional feature image of the target space is generated through the two-dimensional feature extraction model; according to the two-dimensional feature image including adjacent information between the first voxels and the second voxels, object information in the target space is determined.

[0258] In one example, the device is a vehicle or a road test perception device, and the device may also include a three-dimensional space scanning device for collecting environmental point cloud data.

[0259] In another example, the device is a server or a personal computer, etc., and the device can also be used to receive object detection requests for environmental point cloud data sent by vehicles or road test perception devices, and return detected object information to the requesting party.

[0260] Fourth embodiment

[0261] In the above embodiment, an object detection method is provided, and correspondingly, the present application also provides a vehicle driving information determination method. The execution subject of the method includes but is not limited to an unmanned vehicle, and may also be other equipment such as a road test perception device.

[0262] Please see Figure 7 , which is a flow chart of an embodiment of a method for determining vehicle driving information of the present application. A method for determining vehicle driving information provided by the present application includes:

[0263] Step S701: Determine, based on the environmental point cloud data, first feature data of multiple first voxels corresponding to a first spatial resolution in the target space and second feature data of multiple second voxels in the first voxels corresponding to at least one second spatial resolution, and use the first feature data and the second feature data as the third feature data of the first voxel.

[0264] Step S703: using the third feature data of the first voxel as input data of a two-dimensional feature extraction model corresponding to the network depth and the first spatial resolution, and generating a two-dimensional feature image of the target space through the two-dimensional feature extraction model.

[0265] Step S705: determining object information in the target space according to the two-dimensional feature image including the first voxel-to-voxel neighboring information and the second voxel-to-voxel neighboring information.

[0266] The object information may include the location information of the first vehicle, such as the latitude and longitude information of the vehicle, etc., and the information is used as the current location information. The first vehicle includes vehicles other than the execution subject of the method.

[0267] Step S707: Determine the driving speed and / or driving trajectory of the first vehicle based on the current position information and historical position information of the first vehicle.

[0268] The historical position information may be the position information of the first vehicle determined according to the previous frame or previous frames of environmental point cloud data using the method provided in the embodiment of the present application, or may be the vehicle position information determined by other existing positioning technologies, and so on.

[0269] After obtaining the current position information and the historical position information of the first vehicle, the driving speed and / or driving track of the first vehicle can be determined based on the two position information. For example, the distance between the two positions is divided by the time difference between the two frames of data to obtain the driving speed. For another example, based on the two positions, it can be determined that the vehicle's driving track is traveling in a southeast direction, or that the vehicle's driving track is traveling along a certain street, so as to facilitate tracking of the first vehicle, etc.

[0270] In one example, the method may further include the following step: adjusting the driving mode of the second vehicle according to the driving speed and / or driving trajectory. The second vehicle includes a vehicle that executes the method.

[0271] For example, if it is determined based on the current speed and historical speed of the first vehicle that the first vehicle is in an accelerating state and the first vehicle is in front of the second vehicle, the second vehicle can also accelerate appropriately; if the first vehicle is behind the second vehicle and is in an accelerating state, the second vehicle can also accelerate appropriately to avoid vehicle collision and improve the safety of autonomous driving.

[0272] For another example, if it is determined that the driving trajectory of the first vehicle is traveling in a southeast direction, the driving defense line of the second vehicle can be adjusted to facilitate tracking of the first vehicle, and so on.

[0273] It can be seen from the above embodiments that the vehicle driving information determination method provided in the embodiments of the present application determines, based on the environmental point cloud data, first feature data of multiple first voxels corresponding to the first spatial resolution in the target space and second feature data of multiple second voxels in the first voxel corresponding to at least one second spatial resolution respectively, and uses the first feature data and the second feature data as the third feature data of the first voxel; uses the third feature data of the first voxel as input data of a two-dimensional feature extraction model corresponding to the network depth and the first spatial resolution, and generates a two-dimensional feature image of the target space through the two-dimensional feature extraction model; determines object information in the target space based on the two-dimensional feature image including adjacent information between the first voxels and the second voxels; the object information The information includes the current position information of the first vehicle; the driving speed and / or driving trajectory of the first vehicle is determined according to the current position information and historical position information of the first vehicle; this processing method enables three-dimensional image dimensionality reduction processing based on coarse-grained voxels to generate a two-dimensional feature image, and the coarse-grained voxels are divided more finely, so that the feature data of the coarse-grained voxels not only include the voxel feature data of the coarse-grained voxels themselves, but also include the voxel feature data of the fine-grained voxels therein, so that more sufficient point cloud original information is retained in the feature data of the coarse-grained voxels, thereby avoiding the loss of part of the vehicle information, and determining the vehicle's driving speed and / or driving trajectory based on more accurate vehicle information; therefore, it is possible to effectively take into account both higher vehicle driving information accuracy and faster vehicle driving information determination speed.

[0274] Fifth embodiment

[0275] In the above-mentioned embodiment, a method for determining vehicle driving information is provided. Correspondingly, the present application also provides a device for determining vehicle driving information. The device for determining vehicle driving information provided by the present application includes:

[0276] a feature data determination unit, configured to determine, based on the environmental point cloud data, first feature data of a plurality of first voxels corresponding to a first spatial resolution in a target space and second feature data of a plurality of second voxels in the first voxels corresponding to at least one second spatial resolution, respectively, and use the first feature data and the second feature data as third feature data of the first voxels;

[0277] a two-dimensional feature extraction unit, configured to use the third feature data of the first voxel as input data of a two-dimensional feature extraction model corresponding to a network depth and a first spatial resolution, and generate a two-dimensional feature image of a target space through the two-dimensional feature extraction model;

[0278] A position information determining unit, configured to determine object information in the target space according to the two-dimensional feature image including the first voxel-to-voxel adjacent information and the second voxel-to-voxel adjacent information; the object information includes the current position information of the first vehicle;

[0279] The driving information determination unit is used to determine the driving speed and / or driving trajectory of the first vehicle based on the current position information and historical position information of the first vehicle.

[0280] Optionally, the device further comprises:

[0281] The vehicle driving mode adjustment unit is used to adjust the driving mode of the second vehicle according to the driving speed and / or driving trajectory.

[0282] Sixth embodiment

[0283] In the above-mentioned embodiment, a method for determining vehicle driving information is provided. Correspondingly, the present application also provides a device. The device provided by the present application includes:

[0284] Processor; and

[0285] A memory is used to store a program for implementing a method for determining vehicle driving information. After the device is powered on and runs the program of the method for determining vehicle driving information through the processor, the following steps are performed: first feature data of multiple first voxels corresponding to a first spatial resolution in a target space and second feature data of multiple second voxels in the first voxel corresponding to at least one second spatial resolution respectively are determined based on environmental point cloud data, and the first feature data and the second feature data are used as third feature data of the first voxel; the third feature data of the first voxel is used as input data of a two-dimensional feature extraction model corresponding to a network depth and the first spatial resolution, and a two-dimensional feature image of the target space is generated through the two-dimensional feature extraction model; object information in the target space is determined based on the two-dimensional feature image including adjacent information between first voxels and adjacent information between second voxels; the object information includes current position information of the first vehicle; and the driving speed and / or driving trajectory of the first vehicle is determined based on the current position information and historical position information of the first vehicle.

[0286] Seventh embodiment

[0287] In the above embodiment, an object detection method is provided. Correspondingly, the present application also provides an object detection method. The execution subject of the method includes but is not limited to an unmanned vehicle, and may also be other equipment such as a road test perception device.

[0288] Please see Figure 8 , which is a flow chart of an embodiment of an object detection method of the present application. An object detection method provided by the present application includes:

[0289] Step S801: Determine, based on the environmental point cloud data, first feature data of multiple first voxels corresponding to a first spatial resolution in the target space and second feature data of multiple second voxels in the first voxel corresponding to at least one second spatial resolution, and use the first feature data and the second feature data as the third feature data of the first voxel.

[0290] Step S803: using the third feature data of the first voxel as input data of a two-dimensional feature extraction model corresponding to the network depth and the first spatial resolution, and generating a two-dimensional feature image of the target space through the two-dimensional feature extraction model.

[0291] Step S805: determining first object information in the target space according to the two-dimensional feature image including the first voxel-to-voxel neighboring information and the second voxel-to-voxel neighboring information.

[0292] Step S807: Determine second object information in the target space according to the environmental perception data outside the environmental point cloud data and the first object information.

[0293] The perception data includes other types of environmental perception data besides environmental point cloud data, including but not limited to at least one of the following data: two-dimensional image, millimeter wave radar perception data.

[0294] For example, the execution subject of the method is an unmanned vehicle, which is equipped with a laser radar, a millimeter-wave radar and a camera. These three types of sensors collect different types of road environment perception data respectively. The vehicle processes various perception data separately, wherein the first object information is determined from the point cloud data through the above steps 701 to 705, and the third object information is determined from the two-dimensional image through other existing technologies, and the fourth object information is determined from the millimeter-wave radar perception data, and then the third object information and the fourth object information are combined to adjust the first object information, and the adjusted object information is called the second object information. It can be seen that the second object information is more accurate than the other three object information, and the target confidence is higher.

[0295] In one example, the method may further include the following steps:

[0296] The second object information is sent to surrounding vehicles through short-distance communication, and the surrounding vehicles can adjust their driving methods and the like according to the second object information.

[0297] It can be seen from the above embodiments that the object detection method provided in the embodiments of the present application determines, based on the environmental point cloud data, first feature data of multiple first voxels corresponding to the first spatial resolution in the target space and second feature data of multiple second voxels in the first voxel corresponding to at least one second spatial resolution respectively, and uses the first feature data and the second feature data as the third feature data of the first voxel; uses the third feature data of the first voxel as input data of a two-dimensional feature extraction model corresponding to the network depth and the first spatial resolution, and generates a two-dimensional feature image of the target space through the two-dimensional feature extraction model; determines the first object information in the target space based on the two-dimensional feature image including the adjacent information between the first voxels and the second voxel; and determines the first object information in the target space based on the environmental point cloud data. The environmental perception data outside the data and the first object information are used to determine the second object information in the target space; this processing method enables three-dimensional image dimensionality reduction processing based on coarse-grained voxels to generate a two-dimensional feature image, and the coarse-grained voxels are divided more finely, so that the feature data of the coarse-grained voxels not only include the voxel feature data of the coarse-grained voxels themselves, but also include the voxel feature data of the fine-grained voxels therein, so that more sufficient original information of the point cloud is retained in the feature data of the coarse-grained voxels, thereby avoiding the loss of part of the object information, and based on the first object information, further according to other perception data other than the point cloud (such as two-dimensional images, millimeter-wave radar perception data, etc.) to determine more accurate second object information; therefore, it can effectively take into account both higher object detection accuracy and faster detection speed.

[0298] Eighth embodiment

[0299] In the above-mentioned embodiment, an object detection method is provided. Correspondingly, the present application also provides an object detection device. The object detection device provided by the present application includes:

[0300] a feature data determination unit, configured to determine, based on the environmental point cloud data, first feature data of a plurality of first voxels corresponding to a first spatial resolution in a target space and second feature data of a plurality of second voxels in the first voxels corresponding to at least one second spatial resolution, respectively, and use the first feature data and the second feature data as third feature data of the first voxels;

[0301] a two-dimensional feature extraction unit, configured to use the third feature data of the first voxel as input data of a two-dimensional feature extraction model corresponding to a network depth and a first spatial resolution, and generate a two-dimensional feature image of a target space through the two-dimensional feature extraction model;

[0302] A first object information determining unit, configured to determine first object information in a target space according to a two-dimensional feature image including first voxel-to-voxel neighboring information and second voxel-to-voxel neighboring information;

[0303] The second object information determining unit is used to determine the second object information in the target space according to the environmental perception data outside the environmental point cloud data and the first object information.

[0304] Ninth embodiment

[0305] In the above embodiment, an object detection method is provided, and correspondingly, the present application also provides a device. The device provided by the present application includes:

[0306] Processor; and

[0307] A memory is used to store a program for implementing an object detection method. After the device is powered on and runs the program of the object detection method through the processor, the following steps are performed: first feature data of multiple first voxels corresponding to a first spatial resolution in a target space and second feature data of multiple second voxels in the first voxel corresponding to at least one second spatial resolution respectively are determined according to the environmental point cloud data, and the first feature data and the second feature data are used as third feature data of the first voxel; the third feature data of the first voxel is used as input data of a two-dimensional feature extraction model corresponding to a network depth and the first spatial resolution, and a two-dimensional feature image of the target space is generated through the two-dimensional feature extraction model; first object information in the target space is determined according to the two-dimensional feature image including adjacent information between first voxels and adjacent information between second voxels; second object information in the target space is determined according to environmental perception data outside the environmental point cloud data and the first object information.

[0308] Tenth embodiment

[0309] In the above embodiment, an object detection method is provided. Correspondingly, the present application also provides an object detection method. The execution subject of the method includes but is not limited to an unmanned vehicle, and may also be other equipment such as a road test perception device.

[0310] Please see Fig. 9 , which is a flow chart of an embodiment of an object detection method of the present application. An object detection method provided by the present application includes:

[0311] Step S901: Acquire a two-dimensional environment image of the target space.

[0312] For example, an unmanned vehicle collects a two-dimensional environmental image of the driving road through an image acquisition device (such as a camera, etc.) installed on it.

[0313] Step S903: constructing a three-dimensional environment image of the target space according to the two-dimensional environment image.

[0314] To construct a three-dimensional image from a two-dimensional image, relatively mature existing technologies can be used, such as using binocular stereo vision to estimate depth, thereby obtaining a three-dimensional depth image, etc. Since the method of constructing a three-dimensional image from a two-dimensional image belongs to a relatively mature existing technology, it will not be described here.

[0315] It should be noted that the acquired two-dimensional environment image of the target space can be a single image or multiple images. To construct a three-dimensional environment image of the target space based on multiple two-dimensional environment images, strict internal and external parameters of multiple image cameras need to be calibrated.

[0316] Step S905: According to the three-dimensional environment image, determine the first feature data of multiple first voxels corresponding to the first spatial resolution in the target space and the second feature data of multiple second voxels in the first voxels corresponding to at least one second spatial resolution, and use the first feature data and the second feature data as the third feature data of the first voxel.

[0317] In this embodiment, based on point data on the surface of the three-dimensional environment image, first feature data of multiple first voxels corresponding to the first spatial resolution in the target space and second feature data of multiple second voxels in the first voxel corresponding to at least one second spatial resolution are determined, and the first feature data and the second feature data are used as the third feature data of the first voxel.

[0318] Step S907: using the third feature data of the first voxel as input data of a two-dimensional feature extraction model corresponding to the network depth and the first spatial resolution, and generating a two-dimensional feature image of the target space through the two-dimensional feature extraction model.

[0319] Step S909: determining object information in the target space according to the two-dimensional feature image including the first voxel-to-voxel neighboring information and the second voxel-to-voxel neighboring information.

[0320] It can be seen from the above embodiments that the object detection method provided in the embodiments of the present application obtains a two-dimensional environment image of the target space; constructs a three-dimensional environment image of the target space according to the two-dimensional environment image; determines first feature data of multiple first voxels corresponding to a first spatial resolution in the target space and second feature data of multiple second voxels in the first voxel corresponding to at least one second spatial resolution respectively according to the three-dimensional environment image, and uses the first feature data and the second feature data as third feature data of the first voxel; uses the third feature data of the first voxel as input data of a two-dimensional feature extraction model corresponding to the network depth and the first spatial resolution, and extracts the three-dimensional feature data of the first voxel through the two-dimensional environment image. The feature extraction model generates a two-dimensional feature image of the target space; according to the two-dimensional feature image including the adjacent information between the first voxels and the adjacent information between the second voxels, the object information in the target space is determined; this processing method enables the three-dimensional image to be reduced in dimension based on the coarse-grained voxels to generate a two-dimensional feature image, and the coarse-grained voxels are divided more finely, so that the feature data of the coarse-grained voxels not only includes the voxel feature data of the coarse-grained voxels themselves, but also includes the voxel feature data of the fine-grained voxels therein, so that more sufficient spatial information is retained in the feature data of the coarse-grained voxels, thereby avoiding the loss of some object information; therefore, it is possible to effectively take into account both higher object detection accuracy and faster detection speed.

[0321] In addition, compared with the technical solution of the first embodiment, the processing method adopted in this embodiment does not need to collect point cloud data through lidar, but can directly determine object information based on the two-dimensional image collected by the camera, thereby effectively reducing equipment costs.

[0322] In addition, compared with the technical solution of the prior art that directly determines object information based on the collected two-dimensional image through an object detection model (such as the RefineDet method based on deep learning, etc.), the processing method adopted in this embodiment makes the input data (two-dimensional feature image) of the object detection model include spatial information, and the object information determined based on the two-dimensional feature image is more accurate.

[0323] Eleventh Embodiment

[0324] In the above-mentioned embodiment, an object detection method is provided. Correspondingly, the present application also provides an object detection device. The object detection device provided by the present application includes:

[0325] A two-dimensional image acquisition unit, used to acquire a two-dimensional environment image of the target space;

[0326] A three-dimensional image construction unit, used to construct a three-dimensional environment image of the target space according to the two-dimensional environment image;

[0327] a feature data determining unit, configured to determine, based on the three-dimensional environment image, first feature data of a plurality of first voxels corresponding to a first spatial resolution in a target space and second feature data of a plurality of second voxels in the first voxels corresponding to at least one second spatial resolution, respectively, and use the first feature data and the second feature data as third feature data of the first voxels;

[0328] a two-dimensional feature extraction unit, configured to use the third feature data of the first voxel as input data of a two-dimensional feature extraction model corresponding to a network depth and a first spatial resolution, and generate a two-dimensional feature image of a target space through the two-dimensional feature extraction model;

[0329] The object information determining unit is used to determine the object information in the target space according to the two-dimensional feature image including the first voxel-to-voxel neighboring information and the second voxel-to-voxel neighboring information.

[0330] Twelfth Embodiment

[0331] In the above embodiment, an object detection method is provided, and correspondingly, the present application also provides a device. The device provided by the present application includes:

[0332] Processor; and

[0333] A memory is used to store a program for implementing an object detection method. After the device is powered on and the program of the object detection method is run through the processor, the following steps are performed: acquiring a two-dimensional environment image of a target space; constructing a three-dimensional environment image of the target space according to the two-dimensional environment image; determining first feature data of a plurality of first voxels corresponding to a first spatial resolution in the target space and second feature data of a plurality of second voxels in the first voxel corresponding to at least one second spatial resolution respectively according to the three-dimensional environment image, and using the first feature data and the second feature data as third feature data of the first voxel; using the third feature data of the first voxel as input data of a two-dimensional feature extraction model corresponding to a network depth and a first spatial resolution, and generating a two-dimensional feature image of the target space through the two-dimensional feature extraction model; and determining object information in the target space according to the two-dimensional feature image including adjacent information between first voxels and adjacent information between second voxels.

[0334] Thirteenth Embodiment

[0335] In the above-mentioned embodiment, a lesion detection method is provided. Correspondingly, the present application also provides a lesion detection method. The execution subject of the method includes but is not limited to CT lesion detection equipment and other equipment.

[0336] Please see Fig.10, which is a flow chart of an embodiment of a lesion detection method of the present application. A lesion detection method provided by the present application comprises:

[0337] Step S1001: Acquire at least one computerized tomography (CT) image of a target site.

[0338] For example, a three-dimensional CT device may be used to perform multi-section scanning on the same part of the human body (target part, such as the head, etc.) to obtain at least one CT image of the target part.

[0339] Step S1003: constructing a three-dimensional image of the target part according to at least one CT image.

[0340] To construct a three-dimensional image of the target part based on at least one CT image, relatively mature existing technologies can be used, such as using binocular stereo vision to perform depth estimation to obtain a three-dimensional depth image, etc. Since the method of constructing a three-dimensional image based on a two-dimensional image belongs to a relatively mature existing technology, it will not be described here.

[0341] Step S1005: Based on the three-dimensional image, determine the first feature data of multiple first voxels corresponding to the first spatial resolution in the target part and the second feature data of multiple second voxels in the first voxel corresponding to at least one second spatial resolution, and use the first feature data and the second feature data as the third feature data of the first voxel.

[0342] After multiple inputs from different slices are expressed in the same three-dimensional space, they are further divided into multiple levels of voxels, and the feature data of the first voxel is determined.

[0343] Step S1007: using the third feature data of the first voxel as input data of a two-dimensional feature extraction model corresponding to the network depth and the first spatial resolution, and generating a two-dimensional feature image of the target part through the two-dimensional feature extraction model.

[0344] Step S1009: determining the lesion information of the target part according to the two-dimensional feature image including the first voxel-to-voxel adjacent information and the second voxel-to-voxel adjacent information.

[0345] The lesion information includes but is not limited to lesion type, lesion location, etc.

[0346] It can be seen from the above embodiments that the lesion detection method provided in the embodiments of the present application obtains at least one electronic computer tomography CT image of the target part; constructs a three-dimensional image of the target part according to the at least one CT image; determines, according to the three-dimensional image, first feature data of multiple first voxels corresponding to a first spatial resolution in the target part and second feature data of multiple second voxels in the first voxel corresponding to at least one second spatial resolution respectively, and uses the first feature data and the second feature data as third feature data of the first voxel; uses the third feature data of the first voxel as input data of a two-dimensional feature extraction model corresponding to the network depth and the first spatial resolution, and extracts the third feature data of the first voxel through the two-dimensional feature extraction model. The feature extraction model generates a two-dimensional feature image of the target part; the lesion information of the target part is determined according to the two-dimensional feature image including the adjacent information between the first voxels and the adjacent information between the second voxels; this processing method performs image dimension reduction processing on the three-dimensional image of the target part based on the coarse-grained voxels to generate a two-dimensional feature image, and divides the coarse-grained voxels more finely, so that the feature data of the coarse-grained voxels not only includes the voxel feature data of the coarse-grained voxels themselves, but also includes the voxel feature data of the fine-grained voxels therein, so that more sufficient spatial information is retained in the feature data of the coarse-grained voxels, thereby avoiding the loss of some lesion information; therefore, it is possible to effectively take into account both higher lesion detection accuracy and faster detection speed.

[0347] Fourteenth Embodiment

[0348] In the above-mentioned embodiment, a lesion detection method is provided. Correspondingly, the present application also provides a lesion detection device. The object detection device provided by the present application includes:

[0349] An image acquisition unit, used to acquire at least one computerized tomography (CT) image of a target site;

[0350] A three-dimensional image construction unit, configured to construct a three-dimensional image of a target part according to at least one CT image;

[0351] a feature data determining unit, configured to determine, based on the three-dimensional image, first feature data of a plurality of first voxels corresponding to a first spatial resolution in the target part and second feature data of a plurality of second voxels in the first voxels corresponding to at least one second spatial resolution, respectively, and use the first feature data and the second feature data as third feature data of the first voxels;

[0352] a two-dimensional feature extraction unit, configured to use the third feature data of the first voxel as input data of a two-dimensional feature extraction model corresponding to a network depth and a first spatial resolution, and generate a two-dimensional feature image of a target part through the two-dimensional feature extraction model;

[0353] The lesion information determining unit is used to determine the lesion information of the target part according to the two-dimensional feature image including the first voxel adjacent information and the second voxel adjacent information.

[0354] Fifteenth Embodiment

[0355] In the above-mentioned embodiment, a lesion detection method is provided, and correspondingly, the present application also provides a device. The device provided by the present application includes:

[0356] Processor; and

[0357] A memory is used to store a program for implementing a lesion detection method. After the device is powered on and the program of the lesion detection method is run through the processor, the following steps are performed: acquiring at least one electronic computer tomography (CT) image of the target part; constructing a three-dimensional image of the target part based on the at least one CT image; determining first feature data of a plurality of first voxels corresponding to a first spatial resolution in the target part and second feature data of a plurality of second voxels in the first voxel corresponding to at least one second spatial resolution respectively based on the three-dimensional image, and using the first feature data and the second feature data as third feature data of the first voxel; using the third feature data of the first voxel as input data of a two-dimensional feature extraction model corresponding to a network depth and a first spatial resolution, and generating a two-dimensional feature image of the target part through the two-dimensional feature extraction model; determining lesion information of the target part based on the two-dimensional feature image including adjacent information between first voxels and adjacent information between second voxels.

[0358] Sixteenth Embodiment

[0359] In the above embodiment, an object detection method is provided, and correspondingly, the present application also provides a virtual shopping method. The execution subject of the method includes but is not limited to virtual shopping devices and the like.

[0360] Please see Fig.11 , which is a flow chart of an embodiment of a virtual shopping method of the present application. A virtual shopping method provided by the present application includes:

[0361] Step S1101: Determine, based on the shopping environment point cloud data, first feature data of multiple first voxels corresponding to a first spatial resolution in the shopping space and second feature data of multiple second voxels in the first voxels corresponding to at least one second spatial resolution, and use the first feature data and the second feature data as the third feature data of the first voxel.

[0362] For example, a user can use AR (Augmented Reality) smart glasses to construct a three-dimensional image of the shopping space. Based on the three-dimensional image, the shopping space can be divided into multiple levels of voxels to obtain third feature data.

[0363] In specific implementation, AR glasses can estimate the 3D spatial image of the current shopping scene through binocular vision technology or monocular 3D technology, and obtain a 3D point cloud map by discretizing the 3D image. On this basis, multi-level convolutional networks can be used to detect and identify the target of interest and estimate the position and size of the target.

[0364] Step S1103: using the third feature data of the first voxel as input data of a two-dimensional feature extraction model corresponding to the network depth and the first spatial resolution, and generating a two-dimensional feature image of the shopping space through the two-dimensional feature extraction model.

[0365] Step S1105: determining object information in the shopping space according to the two-dimensional feature image including the first voxel-to-voxel neighboring information and the second voxel-to-voxel neighboring information.

[0366] For example, if the shopping space is an air conditioner sales area, the object information includes various displayed air conditioner models and accurate spatial 3D position information.

[0367] Step S1107: placing a virtual object into the three-dimensional image of the shopping space according to the object information.

[0368] After the object information in the shopping space is identified in the previous step, virtual objects can be added to the three-dimensional image of the space. The virtual objects may include virtual shopping guides, air conditioner sales information, and the like.

[0369] In this embodiment, by identifying the model and 3D spatial position of the target, the corresponding customized 3D model of the target (such as an air conditioner) is placed in the virtual space during virtual shopping, and the user can further interact with the virtual model, such as experiencing opening a refrigerator door.

[0370] It can be seen from the above embodiments that the virtual shopping method provided by the embodiments of the present application determines, based on the shopping environment point cloud data, first feature data of multiple first voxels corresponding to the first spatial resolution in the shopping space and second feature data of multiple second voxels in the first voxel corresponding to at least one second spatial resolution respectively, and uses the first feature data and the second feature data as the third feature data of the first voxel; uses the third feature data of the first voxel as input data of a two-dimensional feature extraction model corresponding to the network depth and the first spatial resolution, and generates a two-dimensional feature image of the shopping space through the two-dimensional feature extraction model; and generates a two-dimensional feature image of the shopping space according to the two-dimensional feature extraction model including the adjacent information between the first voxels and the adjacent information between the second voxels. dimensional feature image to determine the object information in the shopping space; according to the object information, the virtual object is placed in the three-dimensional image of the shopping space; this processing method performs image dimensionality reduction processing on the three-dimensional image of the target shopping space based on the coarse-grained voxels to generate a two-dimensional feature image, and divides the coarse-grained voxels more finely, so that the feature data of the coarse-grained voxels not only includes the voxel feature data of the coarse-grained voxels themselves, but also includes the voxel feature data of the fine-grained voxels therein, so that more sufficient spatial information is retained in the feature data of the coarse-grained voxels, thereby avoiding the loss of some product information; therefore, it is possible to effectively take into account both higher object detection accuracy and faster detection speed, thereby improving product sales rate and user experience.

[0371] Seventeenth Embodiment

[0372] In the above-mentioned embodiment, a virtual shopping method is provided. Correspondingly, the present application also provides a virtual shopping device. The virtual shopping device provided by the present application includes:

[0373] a feature data determining unit, configured to determine, based on the shopping environment point cloud data, first feature data of a plurality of first voxels corresponding to a first spatial resolution in the shopping space and second feature data of a plurality of second voxels in the first voxels corresponding to at least one second spatial resolution respectively, and use the first feature data and the second feature data as third feature data of the first voxels;

[0374] a two-dimensional feature extraction unit, configured to use the third feature data of the first voxel as input data of a two-dimensional feature extraction model corresponding to a network depth and a first spatial resolution, and generate a two-dimensional feature image of the shopping space through the two-dimensional feature extraction model;

[0375] An object information determination unit, configured to determine object information in the shopping space based on a two-dimensional feature image including first voxel-to-voxel neighboring information and second voxel-to-voxel neighboring information;

[0376] The virtual object adding unit is used to place the virtual object into the three-dimensional image of the shopping space according to the object information.

[0377] Eighteenth Embodiment

[0378] In the above embodiment, a virtual shopping method is provided, and correspondingly, the present application also provides a device. The device provided by the present application includes:

[0379] Processor; and

[0380] A memory is used to store a program for implementing a virtual shopping method. After the device is powered on and runs the program of the virtual shopping method through the processor, the following steps are performed: first feature data of multiple first voxels corresponding to a first spatial resolution in a shopping space and second feature data of multiple second voxels in the first voxel corresponding to at least one second spatial resolution are determined according to shopping environment point cloud data, and the first feature data and the second feature data are used as third feature data of the first voxel; the third feature data of the first voxel is used as input data of a two-dimensional feature extraction model corresponding to a network depth and the first spatial resolution, and a two-dimensional feature image of the shopping space is generated through the two-dimensional feature extraction model; object information in the shopping space is determined according to the two-dimensional feature image including adjacent information between first voxels and adjacent information between second voxels; and virtual objects are placed into the three-dimensional image of the shopping space according to the object information.

[0381] Nineteenth Embodiment

[0382] In the above-mentioned embodiment, an object detection method is provided, and correspondingly, the present application also provides a weather prediction method. The execution subject of the method includes but is not limited to a weather prediction device and the like.

[0383] Please see Fig.12 , which is a flow chart of an embodiment of a weather forecasting method of the present application. A weather forecasting method provided by the present application includes:

[0384] Step S1201: Acquire a three-dimensional spatial image of the target cloud layer.

[0385] Step S1203: Determine, based on the three-dimensional spatial image, first feature data of multiple first voxels corresponding to the first spatial resolution in the target cloud layer and second feature data of multiple second voxels in the first voxels corresponding to at least one second spatial resolution, and use the first feature data and the second feature data as third feature data of the first voxel.

[0386] For example, a 3D spatial image of the cloud layer can be collected by a weather radar, and based on the 3D image, the target cloud layer can be divided into multiple levels of voxels to obtain the third characteristic data.

[0387] Step S1205: using the third feature data of the first voxel as input data of a two-dimensional feature extraction model corresponding to the network depth and the first spatial resolution, and generating a two-dimensional feature image of the target cloud layer through the two-dimensional feature extraction model.

[0388] Step S1207: Determine weather information based on the two-dimensional feature image including the first voxel-to-voxel adjacent information and the second voxel-to-voxel adjacent information.

[0389] The weather information includes, but is not limited to, snow, rain, etc.

[0390] It can be seen from the above embodiments that the weather prediction method provided in the embodiments of the present application collects a three-dimensional spatial image of a target cloud layer; determines, based on the three-dimensional spatial image, first feature data of a plurality of first voxels corresponding to a first spatial resolution in the target cloud layer and second feature data of a plurality of second voxels in the first voxel corresponding to at least one second spatial resolution, respectively, and uses the first feature data and the second feature data as third feature data of the first voxel; uses the third feature data of the first voxel as input data of a two-dimensional feature extraction model corresponding to a network depth and the first spatial resolution, and generates a target cloud layer through the two-dimensional feature extraction model. Two-dimensional feature image; determining weather information according to the two-dimensional feature image including adjacent information between the first voxels and adjacent information between the second voxels; this processing method performs image dimensionality reduction processing on the three-dimensional image of the target cloud layer based on the coarse-grained voxels to generate a two-dimensional feature image, and divides the coarse-grained voxels more finely, so that the feature data of the coarse-grained voxels not only includes the voxel feature data of the coarse-grained voxels themselves, but also includes the voxel feature data of the fine-grained voxels therein, so that more sufficient spatial information is retained in the feature data of the coarse-grained voxels, thereby avoiding the loss of some weather information; therefore, it is possible to effectively take into account both higher weather forecast accuracy and faster detection speed.

[0391] Twentieth Embodiment

[0392] In the above-mentioned embodiment, a weather prediction method is provided. Correspondingly, the present application also provides a weather prediction device. The object detection device provided by the present application includes:

[0393] An image acquisition unit, used for acquiring a three-dimensional spatial image of a target cloud layer;

[0394] a feature data determining unit, configured to determine, based on the three-dimensional spatial image, first feature data of a plurality of first voxels corresponding to a first spatial resolution in a target cloud layer and second feature data of a plurality of second voxels in the first voxels corresponding to at least one second spatial resolution, respectively, and use the first feature data and the second feature data as third feature data of the first voxels;

[0395] a two-dimensional feature extraction unit, configured to use the third feature data of the first voxel as input data of a two-dimensional feature extraction model corresponding to a network depth and a first spatial resolution, and generate a two-dimensional feature image of a target cloud layer through the two-dimensional feature extraction model;

[0396] The weather information determination unit is used to determine the weather information according to the two-dimensional feature image including the first voxel-to-voxel adjacent information and the second voxel-to-voxel adjacent information.

[0397] Twenty-first embodiment

[0398] In the above embodiment, a weather prediction method is provided. Correspondingly, the present application also provides a device. The device provided by the present application includes:

[0399] Processor; and

[0400] A memory is used to store a program for implementing a weather prediction method. After the device is powered on and runs the program of the weather prediction method through the processor, the following steps are performed: acquiring a three-dimensional spatial image of a target cloud layer; determining, based on the three-dimensional spatial image, first feature data of a plurality of first voxels corresponding to a first spatial resolution in the target cloud layer and second feature data of a plurality of second voxels in the first voxel corresponding to at least one second spatial resolution, and using the first feature data and the second feature data as third feature data of the first voxel; using the third feature data of the first voxel as input data of a two-dimensional feature extraction model corresponding to a network depth and a first spatial resolution, and generating a two-dimensional feature image of the target cloud layer through the two-dimensional feature extraction model; and determining weather information based on the two-dimensional feature image including adjacent information between first voxels and adjacent information between second voxels.

[0401] Although the present application is disclosed as above in the form of a preferred embodiment, it is not intended to limit the present application. Any technical personnel in this field may make possible changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be based on the scope defined by the claims of the present application.

[0402] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0403] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0404] 1. Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include non-transitory media such as modulated data signals and carrier waves.

[0405] 2. Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

Claims

1. An object detection method, characterized in that: include: Determine, according to the environmental point cloud data, first feature data of a plurality of first voxels corresponding to a first spatial resolution in the target space and second feature data of a plurality of second voxels in the first voxels corresponding to at least one second spatial resolution respectively, and use the first feature data and the second feature data as third feature data of the first voxel; The second spatial resolution is higher than the first spatial resolution; Using the third feature data of the first voxel as input data of a two-dimensional feature extraction model corresponding to a network depth and a first spatial resolution, and generating a two-dimensional feature image of a target space through the two-dimensional feature extraction model; the two-dimensional feature extraction model includes a three-dimensional convolutional network; According to the two-dimensional feature image including the first voxel-to-voxel neighboring information and the second voxel-to-voxel neighboring information, object information in the target space is determined.

2. The method according to claim 1, characterized in that The first characteristic data and the second characteristic data are determined by the following steps: Dividing the target space into a plurality of first voxels according to the environmental point cloud data; dividing the first voxel into a plurality of second voxels; determining the first feature data according to the point cloud data of the first voxel; And, second feature data is determined according to the point cloud data of the second voxel.

3. The method according to claim 1, characterized in that Also includes: determining a region of interest within the target space; Clear the point cloud data outside the area of ​​interest in the environmental point cloud data.

4. The method according to claim 1, characterized in that The step of generating a two-dimensional feature image of a target space by using the two-dimensional feature extraction model comprises: Split the three-dimensional convolution kernel of the image feature extraction layer based on manifold sparse convolution into multiple single convolution kernels; For the first single convolution kernel arranged before the central single convolution kernel in the image feature extraction layer, if there is an output non-empty voxel j generated by the first single convolution kernel k for the input non-empty voxel i, then a convolution index relationship R (i, k, j) between the input non-empty voxel, the first single convolution kernel and the output non-empty voxel is constructed; For the second single convolution kernel arranged after the central single convolution kernel in the image feature extraction layer, if there is a corresponding relationship R(i, k, j), a convolution index relationship R(j, k, i) between the input non-empty voxel j, the second single convolution kernel k and the output non-empty voxel i is constructed; Construct the convolution index relationship R(i, k, i) between the input non-empty voxel i, the central single convolution kernel k and the output non-empty voxel i; According to the convolution index relationship, the single convolution kernel and the input non-empty voxels, the non-empty voxels output by the manifold sparse convolution-based image feature extraction layer are determined.

5. The method according to claim 1 or 4, characterized in that: The step of generating a two-dimensional feature image of a target space by using the two-dimensional feature extraction model comprises: Split the three-dimensional convolution kernel of the image feature extraction layer based on non-manifold sparse convolution into multiple single convolution kernels; Determine the global index j of the output voxel based on the spatial size, padding and convolution step of the convolution kernel; For an input non-empty voxel i, if there is an output non-empty voxel j generated by a single convolution kernel k, then a convolution index relationship R(i, k, i) between the input non-empty voxel i, the single convolution kernel k and the output non-empty voxel j is constructed; According to the convolution index relationship, the single convolution kernel and the input non-empty voxels, the non-empty voxels output by the image feature extraction layer based on non-manifold sparse convolution are determined.

6. The method according to claim 1, characterized in that The objects include: vehicles, people, and obstacles.

7. The method according to claim 1, characterized in that If the multi-level voxel division condition is met, determining the first feature data and the second feature data according to the environmental point cloud data, and using the first feature data and the second feature data as the third feature data of the first voxel; If the multi-level voxel division condition is not met, the first feature data is determined based on the environmental point cloud data, and the first feature data is used as input data of the two-dimensional feature extraction model, and the object information in the target space is determined based on the two-dimensional feature image including adjacent information between the first voxels.

8. The method according to claim 7, characterized in that The multi-level voxel partitioning condition includes at least one of the following conditions: The object recognition accuracy is greater than the accuracy threshold; Object information includes vehicles, people or obstacles on the road; The resource occupancy rate of the processor executing the method is greater than the occupancy rate threshold.

9. The method according to claim 1, characterized in that: The second characteristic data is determined by the following steps: Dividing the target space into a first region requiring multi-voxel division and a second region requiring no multi-voxel division; Second feature data of a plurality of second voxels respectively corresponding to at least one second spatial resolution in the first voxel within the first region are determined.

10. The method according to claim 9, characterized in that The first area includes an area between the road surface and a spatial position where the spatial height is less than a height threshold.

11. The method according to claim 1, characterized in that: The third characteristic data is determined by the following steps: Determining a region within the first voxel to be subjected to a second voxel division; The first feature data of the first voxel and the second feature data in the region to be subjected to the second voxel division are used as the third feature data.

12. An object detection device, characterized in that: include: a feature data determination unit, configured to determine, based on the environmental point cloud data, first feature data of a plurality of first voxels corresponding to a first spatial resolution in a target space and second feature data of a plurality of second voxels in the first voxels corresponding to at least one second spatial resolution, respectively, and use the first feature data and the second feature data as third feature data of the first voxels; The second spatial resolution is higher than the first spatial resolution; A two-dimensional feature extraction unit, configured to use the third feature data of the first voxel as input data of a two-dimensional feature extraction model corresponding to a network depth and a first spatial resolution, and generate a two-dimensional feature image of a target space through the two-dimensional feature extraction model; the two-dimensional feature extraction model includes a three-dimensional convolutional network; The object information determining unit is used to determine the object information in the target space according to the two-dimensional feature image including the first voxel-to-voxel neighboring information and the second voxel-to-voxel neighboring information.

13. The device according to claim 12, characterized in that Also includes: An area of ​​interest determining unit, used to determine an area of ​​interest in the target space; The data clearing unit is used to clear the point cloud data outside the area of ​​interest in the environmental point cloud data.

14. A device, characterized in that include: processor; as well as A memory, used for storing a program for implementing an object detection method, after the device is powered on and the program of the method is run by the processor, the following steps are performed: first feature data of a plurality of first voxels corresponding to a first spatial resolution in a target space and second feature data of a plurality of second voxels in the first voxels corresponding to at least one second spatial resolution respectively are determined according to the environmental point cloud data, and the first feature data and the second feature data are used as third feature data of the first voxel; The second spatial resolution is higher than the first spatial resolution; the third feature data of the first voxel is used as input data of a two-dimensional feature extraction model corresponding to the network depth and the first spatial resolution, and a two-dimensional feature image of the target space is generated by the two-dimensional feature extraction model; the two-dimensional feature extraction model includes a three-dimensional convolutional network; object information in the target space is determined based on the two-dimensional feature image including adjacent information between the first voxels and adjacent information between the second voxels.

15. The device according to claim 14, characterized in that The equipment includes a vehicle; The vehicle includes a three-dimensional space scanning device.

16. A method for determining vehicle driving information, characterized in that: include: Determine, according to the environmental point cloud data, first feature data of a plurality of first voxels corresponding to a first spatial resolution in the target space and second feature data of a plurality of second voxels in the first voxels corresponding to at least one second spatial resolution respectively, and use the first feature data and the second feature data as third feature data of the first voxel; The second spatial resolution is higher than the first spatial resolution; Using the third feature data of the first voxel as input data of a two-dimensional feature extraction model corresponding to a network depth and a first spatial resolution, and generating a two-dimensional feature image of a target space through the two-dimensional feature extraction model; the two-dimensional feature extraction model includes a three-dimensional convolutional network; Determine object information in the target space according to the two-dimensional feature image including the first voxel-to-voxel adjacent information and the second voxel-to-voxel adjacent information; the object information includes the current position information of the first vehicle; The driving speed and / or driving track of the first vehicle is determined according to the current position information and the historical position information of the first vehicle.

17. The method according to claim 16, characterized in that Also includes: The driving mode of the second vehicle is adjusted according to the driving speed and / or the driving trajectory.

18. An object detection method, characterized in that: include: Determine, according to the environmental point cloud data, first feature data of a plurality of first voxels corresponding to a first spatial resolution in the target space and second feature data of a plurality of second voxels in the first voxels corresponding to at least one second spatial resolution respectively, and use the first feature data and the second feature data as third feature data of the first voxel; The second spatial resolution is higher than the first spatial resolution; Using the third feature data of the first voxel as input data of a two-dimensional feature extraction model corresponding to a network depth and a first spatial resolution, and generating a two-dimensional feature image of a target space through the two-dimensional feature extraction model; the two-dimensional feature extraction model includes a three-dimensional convolutional network; Determine first object information in the target space according to the two-dimensional feature image including first voxel-to-voxel neighboring information and second voxel-to-voxel neighboring information; Second object information in the target space is determined based on the environmental perception data outside the environmental point cloud data and the first object information.

19. The method according to claim 18, characterized in that The perception data includes at least one of the following data: a two-dimensional image, and millimeter wave radar perception data.

20. A vehicle driving information determination device, characterized in that: include: a feature data determination unit, configured to determine, based on the environmental point cloud data, first feature data of a plurality of first voxels corresponding to a first spatial resolution in a target space and second feature data of a plurality of second voxels in the first voxels corresponding to at least one second spatial resolution, respectively, and use the first feature data and the second feature data as third feature data of the first voxels; The second spatial resolution is higher than the first spatial resolution; A two-dimensional feature extraction unit, configured to use the third feature data of the first voxel as input data of a two-dimensional feature extraction model corresponding to a network depth and a first spatial resolution, and generate a two-dimensional feature image of a target space through the two-dimensional feature extraction model; the two-dimensional feature extraction model includes a three-dimensional convolutional network; A position information determining unit, configured to determine object information in the target space according to the two-dimensional feature image including the first voxel-to-voxel adjacent information and the second voxel-to-voxel adjacent information; the object information includes the current position information of the first vehicle; The driving information determination unit is used to determine the driving speed and / or driving trajectory of the first vehicle based on the current position information and historical position information of the first vehicle.

21. An object detection device, characterized in that: include: a feature data determination unit, configured to determine, based on the environmental point cloud data, first feature data of a plurality of first voxels corresponding to a first spatial resolution in a target space and second feature data of a plurality of second voxels in the first voxels corresponding to at least one second spatial resolution, respectively, and use the first feature data and the second feature data as third feature data of the first voxels; The second spatial resolution is higher than the first spatial resolution; A two-dimensional feature extraction unit, configured to use the third feature data of the first voxel as input data of a two-dimensional feature extraction model corresponding to a network depth and a first spatial resolution, and generate a two-dimensional feature image of a target space through the two-dimensional feature extraction model; the two-dimensional feature extraction model includes a three-dimensional convolutional network; A first object information determining unit, configured to determine first object information in a target space according to a two-dimensional feature image including first voxel-to-voxel neighboring information and second voxel-to-voxel neighboring information; The second object information determining unit is used to determine the second object information in the target space according to the environmental perception data outside the environmental point cloud data and the first object information.

22. A device, characterized in that include: processor; as well as A memory, used for storing a program for implementing a method for determining vehicle driving information. After the device is powered on and the program of the method is run by the processor, the following steps are performed: first feature data of a plurality of first voxels corresponding to a first spatial resolution in a target space and second feature data of a plurality of second voxels in the first voxels corresponding to at least one second spatial resolution are determined according to the environmental point cloud data, and the first feature data and the second feature data are used as third feature data of the first voxel; The second spatial resolution is higher than the first spatial resolution; the third feature data of the first voxel is used as input data of a two-dimensional feature extraction model corresponding to the network depth and the first spatial resolution, and a two-dimensional feature image of the target space is generated by the two-dimensional feature extraction model; the two-dimensional feature extraction model includes a three-dimensional convolutional network; object information in the target space is determined based on the two-dimensional feature image including adjacent information between the first voxels and adjacent information between the second voxels; the object information includes current position information of the first vehicle; based on the current position information and historical position information of the first vehicle, the driving speed and / or driving trajectory of the first vehicle is determined.

23. A device, characterized in that include: processor; as well as A memory, used for storing a program for implementing an object detection method, after the device is powered on and the program of the method is run by the processor, the following steps are performed: first feature data of a plurality of first voxels corresponding to a first spatial resolution in a target space and second feature data of a plurality of second voxels in the first voxels corresponding to at least one second spatial resolution respectively are determined according to the environmental point cloud data, and the first feature data and the second feature data are used as third feature data of the first voxel; The second spatial resolution is higher than the first spatial resolution; the third feature data of the first voxel is used as input data of a two-dimensional feature extraction model corresponding to the network depth and the first spatial resolution, and a two-dimensional feature image of the target space is generated by the two-dimensional feature extraction model; the two-dimensional feature extraction model includes a three-dimensional convolutional network; according to the two-dimensional feature image including the adjacent information between the first voxels and the adjacent information between the second voxels, the first object information in the target space is determined; Second object information in the target space is determined based on the environmental perception data outside the environmental point cloud data and the first object information.

24. An object detection method, characterized in that: include: Acquire a two-dimensional environment image of the target space; Constructing a three-dimensional environment image of the target space according to the two-dimensional environment image; Determine, according to the three-dimensional environment image, first feature data of a plurality of first voxels corresponding to a first spatial resolution in a target space and second feature data of a plurality of second voxels in the first voxels corresponding to at least one second spatial resolution respectively, and use the first feature data and the second feature data as third feature data of the first voxel; The second spatial resolution is higher than the first spatial resolution; Using the third feature data of the first voxel as input data of a two-dimensional feature extraction model corresponding to a network depth and a first spatial resolution, and generating a two-dimensional feature image of a target space through the two-dimensional feature extraction model; the two-dimensional feature extraction model includes a three-dimensional convolutional network; According to the two-dimensional feature image including the first voxel-to-voxel neighboring information and the second voxel-to-voxel neighboring information, object information in the target space is determined.

25. A method for detecting lesions, characterized in that: include: acquiring at least one computerized tomography (CT) image of the target site; constructing a three-dimensional image of the target site based on at least one CT image; Determine, according to the three-dimensional image, first feature data of a plurality of first voxels corresponding to a first spatial resolution in the target part and second feature data of a plurality of second voxels in the first voxels corresponding to at least one second spatial resolution respectively, and use the first feature data and the second feature data as third feature data of the first voxel; The second spatial resolution is higher than the first spatial resolution; Using the third feature data of the first voxel as input data of a two-dimensional feature extraction model corresponding to a network depth and a first spatial resolution, and generating a two-dimensional feature image of the target part through the two-dimensional feature extraction model; the two-dimensional feature extraction model includes a three-dimensional convolutional network; The lesion information of the target part is determined according to the two-dimensional feature image including the first voxel-to-voxel adjacent information and the second voxel-to-voxel adjacent information.

26. An object detection device, characterized in that: include: A two-dimensional image acquisition unit, used to acquire a two-dimensional environment image of the target space; A three-dimensional image construction unit, used to construct a three-dimensional environment image of the target space according to the two-dimensional environment image; a feature data determining unit, configured to determine, based on the three-dimensional environment image, first feature data of a plurality of first voxels corresponding to a first spatial resolution in a target space and second feature data of a plurality of second voxels in the first voxels corresponding to at least one second spatial resolution, respectively, and use the first feature data and the second feature data as third feature data of the first voxels; The second spatial resolution is higher than the first spatial resolution; A two-dimensional feature extraction unit, configured to use the third feature data of the first voxel as input data of a two-dimensional feature extraction model corresponding to a network depth and a first spatial resolution, and generate a two-dimensional feature image of a target space through the two-dimensional feature extraction model; the two-dimensional feature extraction model includes a three-dimensional convolutional network; The object information determining unit is used to determine the object information in the target space according to the two-dimensional feature image including the first voxel-to-voxel neighboring information and the second voxel-to-voxel neighboring information.

27. A device, characterized in that include: processor; as well as A memory, used to store a program for implementing an object detection method. After the device is powered on and the program of the method is run by the processor, the following steps are performed: acquiring a two-dimensional environment image of a target space; constructing a three-dimensional environment image of the target space according to the two-dimensional environment image; determining first feature data of a plurality of first voxels corresponding to a first spatial resolution in the target space and second feature data of a plurality of second voxels in the first voxels corresponding to at least one second spatial resolution respectively according to the three-dimensional environment image, and using the first feature data and the second feature data as third feature data of the first voxel; The second spatial resolution is higher than the first spatial resolution; the third feature data of the first voxel is used as input data of a two-dimensional feature extraction model corresponding to the network depth and the first spatial resolution, and a two-dimensional feature image of the target space is generated by the two-dimensional feature extraction model; the two-dimensional feature extraction model includes a three-dimensional convolutional network; object information in the target space is determined based on the two-dimensional feature image including adjacent information between the first voxels and adjacent information between the second voxels.

28. A lesion detection device, characterized in that: include: An image acquisition unit, used to acquire at least one computerized tomography (CT) image of a target part; A three-dimensional image construction unit, configured to construct a three-dimensional image of a target part according to at least one CT image; a feature data determining unit, configured to determine, based on the three-dimensional image, first feature data of a plurality of first voxels corresponding to a first spatial resolution in the target part and second feature data of a plurality of second voxels in the first voxels corresponding to at least one second spatial resolution, respectively, and use the first feature data and the second feature data as third feature data of the first voxels; The second spatial resolution is higher than the first spatial resolution; A two-dimensional feature extraction unit, configured to use the third feature data of the first voxel as input data of a two-dimensional feature extraction model corresponding to a network depth and a first spatial resolution, and generate a two-dimensional feature image of a target part through the two-dimensional feature extraction model; the two-dimensional feature extraction model includes a three-dimensional convolutional network; The lesion information determining unit is used to determine the lesion information of the target part according to the two-dimensional feature image including the first voxel adjacent information and the second voxel adjacent information.

29. A device, characterized in that include: processor; as well as A memory, used for storing a program for implementing a lesion detection method, after the device is powered on and the program of the method is run by the processor, the following steps are performed: acquiring at least one electronic computer tomography (CT) image of the target part; constructing a three-dimensional image of the target part according to the at least one CT image; determining, according to the three-dimensional image, first feature data of a plurality of first voxels corresponding to a first spatial resolution in the target part and second feature data of a plurality of second voxels in the first voxels corresponding to at least one second spatial resolution respectively, and using the first feature data and the second feature data as third feature data of the first voxel; The second spatial resolution is higher than the first spatial resolution; the third feature data of the first voxel is used as input data of a two-dimensional feature extraction model corresponding to the network depth and the first spatial resolution, and a two-dimensional feature image of the target part is generated by the two-dimensional feature extraction model; the two-dimensional feature extraction model includes a three-dimensional convolutional network; the lesion information of the target part is determined based on the two-dimensional feature image including adjacent information between the first voxels and adjacent information between the second voxels.

30. A virtual shopping method, characterized in that: include: Determine, according to the shopping environment point cloud data, first feature data of a plurality of first voxels corresponding to a first spatial resolution in the shopping space and second feature data of a plurality of second voxels in the first voxels corresponding to at least one second spatial resolution respectively, and use the first feature data and the second feature data as third feature data of the first voxels; The second spatial resolution is higher than the first spatial resolution; Using the third feature data of the first voxel as input data of a two-dimensional feature extraction model corresponding to a network depth and a first spatial resolution, and generating a two-dimensional feature image of the shopping space through the two-dimensional feature extraction model; the two-dimensional feature extraction model includes a three-dimensional convolutional network; determining object information in the shopping space according to the two-dimensional feature image including the first voxel-to-voxel adjacent information and the second voxel-to-voxel adjacent information; According to the object information, a virtual object is placed into the three-dimensional image of the shopping space.

31. A weather forecasting method, characterized in that: include: Collect three-dimensional spatial images of target clouds; Determine, according to the three-dimensional spatial image, first feature data of a plurality of first voxels corresponding to a first spatial resolution in the target cloud layer and second feature data of a plurality of second voxels in the first voxels corresponding to at least one second spatial resolution respectively, and use the first feature data and the second feature data as third feature data of the first voxel; The second spatial resolution is higher than the first spatial resolution; Using the third feature data of the first voxel as input data of a two-dimensional feature extraction model corresponding to a network depth and a first spatial resolution, and generating a two-dimensional feature image of a target cloud layer through the two-dimensional feature extraction model; the two-dimensional feature extraction model includes a three-dimensional convolutional network; Weather information is determined based on a two-dimensional feature image including first inter-voxel neighboring information and second inter-voxel neighboring information.

32. A virtual shopping device, characterized in that: include: a feature data determining unit, configured to determine, based on the shopping environment point cloud data, first feature data of a plurality of first voxels corresponding to a first spatial resolution in the shopping space and second feature data of a plurality of second voxels in the first voxels corresponding to at least one second spatial resolution respectively, and use the first feature data and the second feature data as third feature data of the first voxels; The second spatial resolution is higher than the first spatial resolution; a two-dimensional feature extraction unit, configured to use the third feature data of the first voxel as input data of a two-dimensional feature extraction model corresponding to a network depth and a first spatial resolution, and generate a two-dimensional feature image of the shopping space through the two-dimensional feature extraction model; the two-dimensional feature extraction model includes a three-dimensional convolutional network; An object information determination unit, configured to determine object information in the shopping space based on a two-dimensional feature image including first voxel-to-voxel neighboring information and second voxel-to-voxel neighboring information; The virtual object adding unit is used to place the virtual object into the three-dimensional image of the shopping space according to the object information.

33. A device, characterized in that include: processor; as well as A memory for storing a program for implementing a virtual shopping method. After the device is powered on and the program of the method is run by the processor, the following steps are performed: first feature data of a plurality of first voxels corresponding to a first spatial resolution in a shopping space and second feature data of a plurality of second voxels in the first voxels corresponding to at least one second spatial resolution are determined according to the shopping environment point cloud data, and the first feature data and the second feature data are used as third feature data of the first voxel; The second spatial resolution is higher than the first spatial resolution; the third feature data of the first voxel is used as input data of a two-dimensional feature extraction model corresponding to the network depth and the first spatial resolution, and a two-dimensional feature image of the shopping space is generated by the two-dimensional feature extraction model; the two-dimensional feature extraction model includes a three-dimensional convolutional network; object information in the shopping space is determined based on the two-dimensional feature image including adjacent information between the first voxels and adjacent information between the second voxels; and a virtual object is placed into the three-dimensional image of the shopping space based on the object information.

34. A weather forecasting device, characterized in that: include: An image acquisition unit, used for acquiring a three-dimensional spatial image of a target cloud layer; a feature data determining unit, configured to determine, based on the three-dimensional spatial image, first feature data of a plurality of first voxels corresponding to a first spatial resolution in a target cloud layer and second feature data of a plurality of second voxels in the first voxels corresponding to at least one second spatial resolution, respectively, and use the first feature data and the second feature data as third feature data of the first voxels; The second spatial resolution is higher than the first spatial resolution; a two-dimensional feature extraction unit, configured to use the third feature data of the first voxel as input data of a two-dimensional feature extraction model corresponding to a network depth and a first spatial resolution, and generate a two-dimensional feature image of a target cloud layer through the two-dimensional feature extraction model; the two-dimensional feature extraction model includes a three-dimensional convolutional network; The weather information determination unit is used to determine the weather information according to the two-dimensional feature image including the first voxel-to-voxel adjacent information and the second voxel-to-voxel adjacent information.

35. A device, characterized in that include: processor; as well as A memory is used to store a program for implementing a weather prediction method. After the device is powered on and the program of the method is run by the processor, the following steps are performed: acquiring a three-dimensional spatial image of a target cloud layer; determining, based on the three-dimensional spatial image, first feature data of a plurality of first voxels corresponding to a first spatial resolution in the target cloud layer and second feature data of a plurality of second voxels in the first voxels corresponding to at least one second spatial resolution respectively, and using the first feature data and the second feature data as third feature data of the first voxel; The second spatial resolution is higher than the first spatial resolution; the third feature data of the first voxel is used as input data of a two-dimensional feature extraction model corresponding to the network depth and the first spatial resolution, and a two-dimensional feature image of the target cloud layer is generated by the two-dimensional feature extraction model; the two-dimensional feature extraction model includes a three-dimensional convolutional network; weather information is determined based on the two-dimensional feature image including adjacent information between the first voxels and adjacent information between the second voxels.

Images