Data fusion method and device for parking and storage medium thereof

Abstract

The application discloses a data fusion method for parking, which comprises the following steps: obtaining a weather condition probability according to weather information of a current position of a vehicle and a collected video segment of a current environment by using a weather classification model, and obtaining a corresponding weather influence factor according to the weather condition probability; and determining a fusion region of visual image data and radar point cloud data which needs to be fused to generate a parking scene image according to a vehicle body length, the weather condition probability and the weather influence factor. The application also provides a device and a storage medium for implementing the method. The application can effectively reduce the operation amount of data fusion.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more particularly to a data fusion method, apparatus and storage medium for parking. Background Technology

[0002] Intelligent driving and intelligent parking have brought convenience to people, but at the same time, how to obtain information about the vehicle's surrounding environment more effectively and accurately has also become a focus of attention.

[0003] In existing technologies, image data is acquired using cameras, and point cloud data is obtained using millimeter-wave radar. The image data and point cloud data are then fused to obtain accurate environmental information such as parking spaces, obstacles, and pedestrians. However, even with the use of image data and point cloud data fusion technology, the computational load of fusion remains high, making it impossible to provide accurate real-time information about the vehicle's surrounding environment for automatic parking. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a data fusion method, apparatus and storage medium for parking, which can reduce the fusion calculation of image data and point cloud data, and is applicable to parking scenarios.

[0005] An embodiment of the present invention provides a data fusion method for parking, executed in a computing device. The method includes: acquiring weather information of the vehicle's current location; acquiring video clips of the current environment; inputting the weather information and the video clips into a weather classification model to obtain weather condition probabilities; acquiring corresponding weather influence factors based on the weather condition probabilities; determining a fusion region for data fusion of visual image data and radar point cloud data based on the vehicle's body length, the weather condition probabilities, and the weather influence factors; and determining whether to perform data fusion of the visual image data and radar point cloud data to generate a parking scene image based on the relationship between the distance information of the radar point cloud data and the fusion region.

[0006] An embodiment of the present invention also provides a data fusion apparatus for parking, including a processor; and a memory for storing a computer program, which, when executed by the processor, causes the processor to implement the data fusion method for parking.

[0007] An embodiment of the present invention also provides a storage medium on which a computer program is stored, wherein the computer program, when executed by a processor, implements the data fusion method for parking.

[0008] Compared with existing technologies, the data fusion method, device and storage medium for parking provided by the present invention can dynamically adjust the fusion area for data fusion of visual image data and radar point cloud data according to the weather conditions at the current location of the vehicle, so as to effectively reduce the amount of computation for data fusion. Attached Figure Description

[0009] Figure 1 This is a flowchart of a data fusion method for parking according to an embodiment of the present invention.

[0010] Figure 2 This is a flowchart of a data fusion-based object detection method according to an embodiment of the present invention.

[0011] Figure 3 This is a block diagram of a data fusion device for parking according to an embodiment of the present invention.

[0012] Explanation of main component symbols Steps S101~S106; S201~S203 Device 300 Processor 302 Memory 304 Sensing unit 306 Communication interface 308 The following detailed description, in conjunction with the accompanying drawings, will further illustrate the present invention. Detailed Implementation

[0013] To facilitate understanding and implementation of this invention by those skilled in the art, the invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that this invention provides many applicable inventive concepts, which can be implemented in various specific forms. Those skilled in the art can utilize the details described in these or other embodiments, as well as other available structural, logical, and electrical variations, to implement the invention without departing from its spirit and scope.

[0014] This specification provides different embodiments to illustrate the technical features of different implementations of the invention. The configuration of elements in the embodiments is for illustrative purposes only and is not intended to limit the invention. Furthermore, the repetition of some reference numerals in the embodiments is for simplification and does not imply any correlation between different embodiments. The same element numbers used in the illustrations and specification represent the same or similar components. The illustrations in this specification are simplified and not drawn to scale.

[0015] Furthermore, in describing some embodiments of the present invention, the specification describes the method and / or procedure of the present invention in a specific order of steps. However, since the method and procedure are not necessarily performed according to the specific order of steps described, they are not limited to the specific order of steps. Those skilled in the art will understand that other orders are also possible implementations. Therefore, the specific order of steps described in the specification is not intended to limit the scope of the patent application. Moreover, the scope of the present invention for the method and / or procedure is not limited to the order of execution steps written therein, and those skilled in the art will understand that adjusting the order of execution steps does not depart from the spirit and scope of the present invention.

[0016] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. The terminology used herein in the specification of this invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items. Some embodiments of the invention are described in detail below with reference to the accompanying drawings.

[0017] Please see Figure 1 The diagram shows a flowchart of a data fusion method for parking according to an embodiment of the present invention.

[0018] Step S101: Obtain weather information for the vehicle's current location.

[0019] Weather information for a vehicle's current location can be obtained through publicly available online data, such as by connecting to weather apps or meteorological websites.

[0020] Step S102: Collect video clips of the current environment.

[0021] In this step, a vehicle-mounted camera can be used to collect video data for a preset duration, such as ten seconds.

[0022] Step S103: Input the weather information and video clips into the weather classification model to obtain the weather condition probability output by the weather classification model.

[0023] In this step, the weather classification model employs a convolutional neural network (CNN) or a deep convolutional neural network (DCNN), such as DenseNet or ResNet, without limitation. The output of the neural network is fed into a softmax classification layer to classify weather conditions and obtain weather condition probabilities. For example, assuming the weather classification model categorizes weather as rainy, sunny, cloudy, foggy, and overcast, the output of the model is a predicted score for each weather category. The output layer is then combined with a softmax function to map the classification output probabilities onto a softmax layer. Between these points, the normalized sum of the classification output probabilities is 1. The final probability of the weather condition corresponding to the vehicle's current location being a specific weather type is called the weather condition probability. In this step, the category with the highest predicted probability is used as the prediction result. For example, after the neural network passes through the softmax layer, the probability of classifying it as rainy (0.7) far surpasses other categories, and the final output weather condition probability will be rainy (0.7).

[0024] Step S104: Obtain the corresponding weather influencing factors based on the probability of weather conditions.

[0025] In this embodiment, each weather category has its own pre-defined weather influencing factor. The weather influencing factor... It can be dynamically adjusted based on the degree to which weather type affects object detection. For example, when the weather type is sunny, weather conditions can be considered not to be a factor affecting object detection; in this case, the weather influence factor is relatively small. When the weather type is severe, such as rain or fog, and weather conditions are a significant factor affecting object detection, the weather influence factor is substantial. It can be set to a value close to 1.

[0026] Step S105: Determine the fusion area for data fusion of visual image data and radar point cloud data based on the vehicle's body length, weather condition probability, and weather influencing factors.

[0027] In this step, the vehicle's body length is The probability of weather conditions is and weather influencing factors Calculate the radius of the first region used to divide the fusion region and the observation region respectively. With the radius of the second region as follows: ; The fusion area is defined as the area centered on the vehicle and having a radius equal to the first area. As the coverage area of ​​the radius, the observation area is the area centered on the vehicle and with a radius of the second region. The radius is the coverage area formed after deducting the fusion region.

[0028] Step S106: Based on the relationship between the distance information of the radar point cloud data and the fusion area, determine whether to perform data fusion of visual image data and radar point cloud data to generate a parking scene image.

[0029] In this step, based on whether the distance information of the radar point cloud data falls within the coverage area of ​​the fusion region, it is determined whether to perform data fusion on the visual image data and the radar point cloud data. The coverage area that falls within the integration zone, that is This requires data fusion of visual image data and radar point cloud data. Conversely, when the distance information of the radar point cloud data... Coverage that does not fall within the fusion area but falls within the observation area, i.e. Therefore, data fusion of visual image data and radar point cloud data is not required. In this embodiment, the data fusion algorithm is not limited. Deep learning methods or the default feature and rule matching method can be used for data fusion. Finally, a parking scene image is generated based on the fusion result, thereby achieving high-precision 3D target detection in the parking space.

[0030] In different embodiments, in addition to the distance information of the radar point cloud data, the speed information of the radar point cloud data can also be used to determine whether data fusion is needed.

[0031] Specifically, when the distance information of radar point cloud data... The coverage area that falls within the integration zone, that is This requires data fusion of visual image data and radar point cloud data. When the distance information from the radar point cloud data... Not falling within the coverage area of ​​the fusion zone, but falling within the coverage area of ​​the observation zone, that is... If the speed information of the radar point cloud data exceeds the default safe speed threshold, then data fusion of visual image data and radar point cloud data is required. Otherwise, when the distance information of the radar point cloud data... It does not fall within the fusion zone, but falls within the observation zone, that is... If the speed information of the radar point cloud data is less than or equal to the default safe speed threshold, then data fusion of visual image data and radar point cloud data is not required.

[0032] In one example, the default safe speed threshold is 10 kilometers per hour. The actual velocity of the object. A positive value indicates the radial velocity observed by radar point cloud data, and the direction is towards the vehicle. A negative value indicates movement away from the vehicle. Therefore, according to... The comparison with the default safe speed threshold can determine whether an object is approaching the car and its speed exceeds the default safe speed threshold, or whether the object and the car are within the safe speed range.

[0033] In step S106, for the area where data fusion is required, object detection of the three-dimensional target can be further performed based on the results of data fusion.

[0034] Please see Figure 2 The diagram shows a flowchart of an object detection method based on data fusion according to an embodiment of the present invention.

[0035] Step S201: Input the visual image data into the object detection model to obtain the 3D detection box of the object.

[0036] In one example, the object detection model can directly obtain the 3D bounding box of an object from visual image data using a neural network. The object detection model can use any known image-based 3D detection model; no specific limitations are imposed here.

[0037] For example, if an object detection model is trained using the CenterNet object detection method, the model can detect the center point, width, and height information of an object. .

[0038] Furthermore, the object's 3D bounding box can be obtained from the object's center point and its width and height information. .

[0039] Step S202: Adjust the 3D detection box of the object according to the probability of weather conditions and weather influence factors to obtain the bounding box of the object.

[0040] Continuing with the previous example, the 3D bounding box of the object is adjusted using weather condition probabilities and weather influence factors as follows: This allows you to obtain the bounding box of the adjusted object.

[0041] Step S203 involves fusing the object's bounding box with the position, height, and velocity of the radar point cloud data to obtain fused object information. This object information includes the object's bounding box, coordinates, shape, and velocity information.

[0042] In one embodiment, the bounding box, coordinates, shape, and velocity information of an object are output to the vehicle. When parking, the vehicle can generate a corresponding avoidance box based on the object's bounding box, coordinates, shape, and velocity information, and display graphic instructions on a display installed on the vehicle to guide the driver to avoid the object when parking.

[0043] In different embodiments, the vehicle-mounted device can generate a parking trajectory based on the vehicle's current position, driving information, the bounding box, coordinates, shape, and speed information of the target parking position and the object, in order to complete automatic parking or display it on a monitor installed on the vehicle to assist the user in autonomous parking control.

[0044] Figure 1 or Figure 2The described method flow can be implemented by a device that is detachably or fixedly mounted on a vehicle. In one embodiment, the device is a computing device, the hardware block diagram of which is shown below. Figure 3 As shown. Those skilled in the art should understand that, Figure 3 The composition of the device 300 shown does not constitute a limitation of the embodiments of the present invention. Figure 3 The device 300 shown is simplified for ease of description, and in different embodiments it may include fewer or more components than shown.

[0045] Please see Figure 3 The diagram shown is a hardware block diagram of a data fusion device 300 for parking according to an embodiment of the present invention. The device 300 includes a processor 302, a memory 304, a sensing unit 306, and a communication interface 308. Figure 1 or Figure 2 The described method flow can be implemented by a device 300 that is detachably or fixedly mounted on a vehicle. Those skilled in the art should understand that... Figure 3 The composition of the device 300 shown does not constitute a limitation of the embodiments of the present invention. Figure 3 The device 300 shown is simplified for ease of description, and in different embodiments it may include fewer or more components than shown.

[0046] In one embodiment, the processor 302 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits packaged with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 302 is the control unit of the device 300, connecting various components of the device 300 via various interfaces and lines. It executes computer programs or modules stored in the memory 302 and calls data stored in the memory 302 to perform various functions of the device 300 and process data, such as a data fusion method for parking.

[0047] In one embodiment, the memory 304 is used to store computer program code and various data, such as data fusion methods for parking, and to enable high-speed, automatic access to programs or data during the operation of the device 300. The memory 304 includes read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable storage medium capable of carrying or storing data.

[0048] In one embodiment, the sensing unit 306 includes several sensors for sensing environmental information about the vehicle's surroundings. For example, the sensing unit 306 may include a positioning system, an inertial measurement unit, a lidar, a 4D millimeter-wave radar, and a camera. The sensing unit 306 may also include sensors for monitoring internal vehicle systems, such as an in-vehicle air quality monitor, a fuel gauge, and an oil temperature gauge. In different embodiments, different sensors in the sensing unit 306 may be different, independent devices connected to the device 300 via wireless or wired communication.

[0049] In one embodiment, the communication interface 308 is composed of a communication circuit for communicating data or information with external devices, such as vehicle computers, in-vehicle computers, etc., also known as vehicle terminals.

[0050] In summary, the data fusion method, apparatus, and storage medium for parking of the present invention can determine the area requiring data fusion by incorporating weather factors, thereby reducing the computational load of data fusion. Simultaneously, when detecting objects, the bounding boxes of the objects can be adjusted based on weather factors, ensuring the accuracy of the fused information and facilitating assisted parking or automatic parking control.

[0051] It is worth noting that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A data fusion method for parking, executed in a computing device, characterized in that, The method includes: Obtain weather information for the vehicle's current location; Capture video clips of the current environment; The weather information and the video clip are input into a weather classification model to obtain the probability of weather conditions. Obtain the corresponding weather influencing factors based on the probability of the described weather conditions; The fusion region for data fusion of visual image data and radar point cloud data is determined based on the vehicle's body length, the probability of the weather conditions, and the weather influencing factors. as well as Based on the relationship between the distance information of the radar point cloud data and the fusion region, it is determined whether to perform data fusion of the visual image data and the radar point cloud data to generate a parking scene image.

2. The data fusion method for parking as described in claim 1, characterized in that, The method of obtaining weather information for the vehicle's current location also includes obtaining the weather information by connecting to a weather application.

3. The data fusion method for parking as described in claim 1, characterized in that, The process of collecting video clips of the current environment also includes: using a vehicle-mounted camera to collect video data of a preset duration.

4. The data fusion method for parking as described in claim 1, characterized in that, The weather classification model uses a neural network model, and the output of the neural network is fed into a softmax classification layer to classify weather conditions and obtain the probability of the weather conditions.

5. The data fusion method for parking as described in claim 4, characterized in that, The weather categories in the softmax classification layer include rainy, sunny, cloudy, foggy, and overcast.

6. The data fusion method for parking as described in claim 5, characterized in that, Each weather category in the weather types is pre-defined with a corresponding weather influence factor, which is used to represent the degree of influence of the weather category on object detection.

7. The data fusion method for parking as described in claim 1, characterized in that, The process of determining the fusion region for data fusion of visual image data and radar point cloud data based on the vehicle's body length, the probability of weather conditions, and the weather influencing factors further includes: The vehicle's body length is described as The probability of the weather conditions is expressed as follows: Description of weather influencing factors The first region radius used to divide the fusion region and the observation region. The calculation formula is With the radius of the second region The calculation formula is The fusion region is defined as the region centered on the vehicle and having a radius equal to the first region. As the coverage area of ​​the radius, the observation area is centered on the vehicle and has a radius of the second area. The radius is the coverage area formed after deducting the fused region.

8. The data fusion method for parking as described in claim 7, characterized in that, The process of determining the fusion region for data fusion of visual image data and radar point cloud data based on the vehicle's body length, the probability of weather conditions, and the weather influencing factors further includes: When the distance information of the radar point cloud data The coverage area falling within the fusion region, that is It is necessary to perform data fusion between the visual image data and the radar point cloud data; When the distance information of the radar point cloud data The coverage area that does not fall within the fusion region but falls within the observation region, i.e. Therefore, data fusion of the visual image data and the radar point cloud data is not required.

9. A data fusion device for parking, characterized in that, include Processor; and A memory for storing a computer program that, when executed by the processor, causes the processor to implement the data fusion method for parking as described in any one of claims 1 to 8.

10. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the data fusion method for parking as described in any one of claims 1 to 8.

Images