Panoramic image compensation method and device based on information fusion, equipment and medium

Abstract

The invention discloses a panoramic image compensation method and device based on information fusion, equipment and a medium. The method comprises the following steps: determining a driving scene according to the driving speed and the driving gear of a vehicle; whether panoramic image compensation is triggered or not is judged according to the collected environment information and obstacle information and the driving scene; if panoramic image compensation is triggered, restoring the obstacle information through a three-dimensional restoration method to obtain an obstacle restoration model; and performing alignment processing on the obstacle restoration model and the obtained obstacle coordinates, and then superposing the obstacle restoration model and the obtained obstacle coordinates to the panoramic image. By implementing the method provided by the invention, the problem that the road scene cannot be intuitively obtained in a complex environment in the prior art can be solved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent driving, and in particular to a panoramic image compensation method, device, equipment and medium based on information fusion. Background Art

[0002] In the current automotive industry, 360-degree panoramic imaging technology has become a standard feature of various models, providing drivers with a full range of vision. However, in complex environments such as night and rainy days, the environmental perception and display capabilities of this technology seem to be insufficient. Modern cars generally adopt a solution that combines 360-degree panoramic imaging with PDC reversing radar. By combining parametric display with auditory assistance, the shortcomings of 360-degree panoramic imaging in visual display are effectively compensated. Despite this, users still find it difficult to intuitively and vividly grasp the specific conditions of the surrounding environment and obstacles when facing complex environments during actual use. Summary of the invention

[0003] The embodiments of the present invention provide a panoramic image compensation method, device, equipment and medium based on information fusion, aiming to solve the problem in the prior art that the road scene cannot be intuitively obtained in a complex environment.

[0004] In a first aspect, an embodiment of the present invention provides a panoramic image compensation method based on information fusion, which includes: determining a driving scene according to a driving speed and a driving gear of a vehicle; judging whether to trigger panoramic image compensation according to collected environmental information and obstacle information and the driving scene; if panoramic image compensation is triggered, restoring the obstacle information through a three-dimensional restoration method to obtain an obstacle restoration model; aligning the obstacle restoration model with the acquired obstacle coordinates and superimposing them on the panoramic image.

[0005] In the second aspect, an embodiment of the present invention also provides a panoramic image compensation device based on information fusion, which includes: a determination unit, used to determine a driving scene according to a driving speed and a driving gear of a vehicle; a judgment unit, used to judge whether to trigger panoramic image compensation according to collected environmental information and obstacle information and the driving scene; a restoration unit, used to restore the obstacle information through a three-dimensional restoration method to obtain an obstacle restoration model if panoramic image compensation is triggered; and an overlay unit, used to align the obstacle restoration model with the acquired obstacle coordinates and then overlay them on the panoramic image.

[0006] In a third aspect, an embodiment of the present invention further provides a computer device, which includes a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the above method when executing the computer program.

[0007] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores a computer program, wherein the computer program includes program instructions, and the program instructions can implement the above method when executed by a processor.

[0008] The embodiment of the present invention provides a panoramic image compensation method, device, equipment and medium based on information fusion. The method includes: determining the driving scene according to the driving speed and driving gear of the vehicle; judging whether to trigger panoramic image compensation according to the collected environmental information and obstacle information and the driving scene; if the panoramic image compensation is triggered, the obstacle information is restored by a three-dimensional restoration method to obtain an obstacle restoration model; the obstacle restoration model is aligned with the obtained obstacle coordinates and then superimposed on the panoramic image. The embodiment of the present invention collects information about obstacles and the environment around the vehicle by setting a variety of sensors to obtain obstacle information and environmental information. According to the environmental information, obstacle information and the driving scene, it can flexibly judge whether panoramic image compensation is needed in the current scene, restore the three-dimensional obstacle restoration model by a three-dimensional restoration method and superimpose it on the panoramic image to compensate for the missing part of the panoramic image, so as to intuitively display the surrounding environment, so that the user can always perceive the surrounding road scene in different complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0010] Figure 1 A schematic diagram of a flow chart of a panoramic image compensation method based on information fusion provided by an embodiment of the present invention;

[0011] Figure 2 A schematic diagram of a sub-process of a panoramic image compensation method based on information fusion provided by an embodiment of the present invention;

[0012] Figure 3 A schematic diagram of a sub-process of a panoramic image compensation method based on information fusion provided by an embodiment of the present invention;

[0013] Figure 4 A schematic diagram of a sub-process of a panoramic image compensation method based on information fusion provided by an embodiment of the present invention;

[0014] Figure 5 A schematic diagram of a sub-process of a panoramic image compensation method based on information fusion provided by an embodiment of the present invention;

[0015] Figure 6 A schematic diagram of a sub-process of a panoramic image compensation method based on information fusion provided by an embodiment of the present invention;

[0016] Figure 7 A panoramic image compensation diagram of a panoramic image compensation method based on information fusion provided by an embodiment of the present invention;

[0017] Figure 8 A schematic diagram of a sub-process of a panoramic image compensation method based on information fusion provided by an embodiment of the present invention;

[0018] Fig. 9 A schematic block diagram of a panoramic image compensation device based on information fusion provided by an embodiment of the present invention;

[0019] Fig.10 A schematic block diagram of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0022] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0023] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0024] See also Figure 1 , Figure 1A flowchart of a panoramic image compensation method based on information fusion provided in an embodiment of the present invention. The panoramic image compensation method based on information fusion in this embodiment can be applied to the 360-degree panoramic image compensation of an automobile system, wherein a number of sensors such as cameras, ultrasonic radars, and lidars are installed around the vehicle, and information data for panoramic image compensation is provided by the sensor devices. According to the information provided by the sensor devices, it can be determined at all times whether the panoramic image compensation is triggered. If triggered, the obstacle can be restored and superimposed on the panoramic image to compensate for the missing part of the panoramic image, so as to directly display the surrounding environment.

[0025] Figure 1 1 is a flow chart of a panoramic image compensation method based on information fusion provided by an embodiment of the present invention. As shown in the figure, the method includes the following steps S110-S140.

[0026] S110: Determine a driving scene according to the driving speed and driving gear of the vehicle.

[0027] In this embodiment, the driving speed is the current driving speed of the vehicle, and the driving gear is the gear selected in the current operation of the vehicle, and the driving gear includes: parking gear (P gear), reverse gear (R gear), forward gear (D gear), etc. The driving scene is the current operation scene of the vehicle, for example, the driving scene includes forward scene, parking scene, etc. The driving scene is determined according to the driving speed and driving gear of the vehicle. Specifically, it is determined whether the driving speed of the vehicle is less than the preset driving speed. In this embodiment, the preset driving speed is 10km / h. If the driving speed is less than the preset driving speed and the current gear is the forward gear (D gear), the current driving scene is determined to be the parking scene. If the current gear is the forward gear (D gear) and the driving speed of the vehicle is greater than the preset driving speed, the current driving scene is determined to be the forward scene. The preset driving speed can be determined according to information such as vehicle body performance, and is not limited to this. In addition, if the current gear information is the reverse gear (R gear), the current driving scene can also be determined to be the parking scene. By determining the current driving scene of the vehicle according to the driving speed and the driving gear, it is convenient to make comprehensive panoramic image compensation judgments according to different driving scenes.

[0028] S120: Determine whether to trigger panoramic image compensation according to the collected environment information and obstacle information and the driving scene.

[0029] In this embodiment, the environmental information and obstacle information are information about the environment and obstacles around the vehicle body collected by sensor devices such as cameras, ultrasonic radars and laser radars pre-installed on the vehicle body. The environmental information includes current light information, rainfall information and other contents, and the obstacle information includes information such as obstacle point cloud data. The sensor device is connected to the intelligent driving domain controller of the vehicle body to transmit the collected data to the inside of the vehicle body. According to the collected environmental information and obstacle information and the driving scene, it is judged whether to trigger panoramic image compensation. Specifically, it can be judged whether the panoramic image is missing due to the light or rainfall information in the environmental information in the current driving scene, and it can also be judged whether the picture is missing according to the distance from the obstacle to judge whether to trigger panoramic image compensation. By judging whether to trigger panoramic image compensation based on the currently collected environmental information and obstacle information and the driving scene, it is possible to judge in real time whether the panoramic image needs to be compensated, so as to ensure that the user can always clearly understand the situation around the vehicle body.

[0030] In one embodiment, if Figure 2 As shown, the step S120 also includes steps S121-S122.

[0031] S121, if the current driving scene is a forward scene, obtaining a driving distance between the vehicle and the obstacle closest to the vehicle;

[0032] S122 determines whether the driving distance is less than a preset safety distance, and if so, determines to trigger panoramic image compensation.

[0033] In this embodiment, the driving scene includes a forward scene. Specifically, if the current driving gear is the forward gear (D gear) and the driving speed is greater than the preset driving speed, the current driving scene is a forward scene. At this time, the SR (Substitutional Reality) restoration interface is displayed by default on the central control display screen of the vehicle, that is, the model map of the corresponding surrounding environment is displayed on the central control display screen, specifically a page showing information such as surrounding vehicles, traffic lights, roads, obstacles, etc. It is convenient for users to drive the vehicle according to the surrounding information. The driving distance between the vehicle and the obstacle closest to it is obtained according to the information collected by the sensors around the vehicle body, and it is determined whether the driving distance is less than the preset safety distance. In this embodiment, the preset safety distance is 60 cm. The preset safety distance can be set according to information such as the vehicle body width, and there is no limitation on this. If the driving distance is less than the preset safety distance, it can be determined that the vehicle enters the narrow road driving assistance mode. Because the driving space is too narrow, the user cannot intuitively view the specific information of the surrounding obstacles, so the default SR restoration interface is automatically switched to a top view of the vehicle model, and a 360-degree real-life image is superimposed, and the distance to the surrounding obstacles is displayed at the same time. Since 360-degree real-life images need to be superimposed, but the images cannot intuitively obtain and display obstacles due to spatial influence, it is necessary to compensate the images according to the specific information of the obstacles. Therefore, when the driving distance is less than the preset safety distance, it is determined to trigger panoramic image compensation. By judging the driving distance between the vehicle and the obstacle closest to it, it is determined whether the current driving road is too narrow to affect the viewing of the surrounding environment, so as to monitor in real time whether the panoramic image needs to be compensated.

[0034] In one embodiment, if Figure 3 As shown, the step S120 also includes steps S123-S125.

[0035] S123, if the current driving scene is a parking scene, determining a parking mode according to the acquired parking activation information;

[0036] S124, if the current parking mode is the manual parking mode, determining whether the panoramic image can feedback a clear image according to the light information and / or the rainfall information;

[0037] S125: If a clear image cannot be fed back, it is determined that panoramic image compensation is triggered.

[0038] In this embodiment, the driving scene includes a parking scene. Specifically, when the driving gear of the vehicle is the reverse gear (R gear), or the forward gear (D gear) but the driving speed is less than the preset driving speed, the current driving scene is a parking scene. When the vehicle is in the parking scene, the car model is rotated to the reverse perspective so that the 360-degree panoramic image interface is displayed on the vehicle central control display screen. Parking activation information is obtained according to the parking information selected by the user. If the parking laser information shows that the user is in manual parking mode, the light information and / or rainfall information in the environmental information is obtained to determine whether the panoramic image can feedback a clear image. Specifically, the light information and rainfall information can be obtained according to the light sensor and rainfall sensor pre-installed on the vehicle body. According to the light information and rainfall information, it is determined whether the current scene is a dark environment scene or a rainy day scene. The dark environment and / or rainy day scene will cause the image collected by the sensor and other equipment to be unclear, and it is impossible to feedback a clear image. Therefore, it can be determined that the current scene triggers panoramic image compensation, and the panoramic image needs to be compensated according to the collected obstacle information. By determining whether to trigger panoramic image compensation according to the light information and / or the rainfall information, the panoramic image is compensated in time to help the user complete manual parking.

[0039] In one embodiment, if Figure 4 As shown, the step S120 also includes steps S126-S127.

[0040] S126, detecting whether the camera signal for feeding back the panoramic image is normal;

[0041] S127: If the camera signal fed back is abnormal, determine to trigger panoramic image compensation.

[0042] In this embodiment, the camera signal is the signal fed back by the camera installed around the vehicle body. The vehicle's CDCU (Central Driving Control Unit) detects whether the fed back camera signal is normal. For example, if the signal fed back by the camera at the rear of the vehicle is not received, the CDCU may determine that the camera signal is abnormal. Because part of the signal is missing, the feedback result of the panoramic image is not clear enough, so it is determined to trigger panoramic image compensation to supplement the missing image in time. It can be understood that whether in the foreground scene, parking scene or other scenes, the camera signal of the panoramic image is always detected. By detecting whether the camera signal used to feedback the panoramic image is normal, it is determined whether to trigger panoramic image compensation, so as to effectively avoid the problem of unclear panoramic image due to missing camera signal.

[0043] In one embodiment, if Figure 5As shown, the step S123 also includes steps S1231-S1234.

[0044] S1231, if the current parking mode is the automatic parking mode, selecting a parking space to be parked according to the parking space information, and obtaining the four corner coordinates of the parking space to be parked, wherein the environmental information includes the parking space information;

[0045] S1232, obtaining a parking area by using the four-corner coordinates through the three-dimensional restoration method;

[0046] S1233, generating a parking path by using a preset path planning method based on the current position information of the vehicle and the position of the parking space to be parked;

[0047] S1234: superimpose the parking area and the parking path into the panoramic image by using a preset fusion method.

[0048] In this embodiment, the automatic parking mode is a mode in which the vehicle parks itself, and the parking space to be parked is a parking space determined according to the parking space information in the environmental information. For example, if a certain parking space in the surrounding area is empty according to the parking space information, then the parking space is the parking space to be parked. The four-corner coordinates of the parking space to be parked are obtained, wherein the vehicle model is taken as the coordinate center, the coordinates of the four corners of the parking space to be parked are obtained, and the four-corner coordinates are used to obtain the parking area through the three-dimensional restoration method, that is, the parking area that fits the three-dimensional scene is generated according to the coordinates of the four corners through the SR restoration method, so that the user can quickly understand the parking space situation. And the parking path from the current position of the vehicle to the position of the parking space to be parked can be generated by the path planning method such as hybridA*, wherein the HybridA* algorithm introduces the actual constraints of the object's motion on the basis of the A algorithm, takes into account the object's motion direction and motion possibility, and thus can perform path planning in an environment with motion constraints. The parking area and the parking path are superimposed on the panoramic image by a preset fusion method. Specifically, the transparency of the parking area and the parking path can be adjusted to 50%, and the parking area and the parking path are superimposed on the panoramic image on the central control display screen. In the automatic parking mode, the parking area and the parking path are superimposed on the panoramic image to intuitively show the parking situation to the user.

[0049] S130: If panoramic image compensation is triggered, the obstacle information is restored by a three-dimensional restoration method to obtain an obstacle restoration model.

[0050] In this embodiment, the three-dimensional restoration method is a method of simulating reality by using three-dimensional graphics technology to generate a three-dimensional visual effect model, for example, an SR restoration method. The obstacle restoration model is a three-dimensional stereoscopic model of the obstacle restored based on the obstacle information. If the panoramic image compensation is triggered, it means that the existing panoramic image cannot clearly show the specific situation of the obstacle to the user. In order to avoid the occurrence of vehicle accidents, the three-dimensional restoration model of the obstacle is displayed in the panoramic image so that the user can have a specific understanding of the obstacle. The obstacle information is restored by a three-dimensional restoration method. Specifically, the type of obstacle and the outline of the obstacle can be restored by an SR restoration method to obtain an obstacle restoration model. By restoring the obstacle restoration model based on the obstacle information and the three-dimensional restoration method, the panoramic image can be compensated by the obstacle restoration model to help the user understand the surrounding scene.

[0051] In one embodiment, if Figure 6 As shown, the step S130 also includes steps S131-S132.

[0052] S131, acquiring point cloud data corresponding to the obstacle information, and performing type recognition according to the point cloud data to acquire the obstacle category;

[0053] S132: performing type matching in a preset obstacle model library according to the obstacle category to obtain the obstacle restoration model.

[0054] In this embodiment, the point cloud data is the data detected and returned by the sensors installed around the vehicle body according to the obstacle information. The intelligent driving domain controller in the vehicle performs data fusion based on the collected point cloud data to obtain the obstacle category, wherein the intelligent driving domain controller is the core component in the intelligent driving system, which undertakes the data processing computing power required for autonomous driving, has powerful algorithms and resources, and can process data of millimeter wave radar, camera, lidar, GPS, inertial navigation and other equipment. According to the obstacle category, type matching is performed in the preset obstacle model library in the central domain controller. For example, if the category of the obstacle is a car, type matching is performed in the preset obstacle model library to obtain a three-dimensional car model, and it is defined as the obstacle restoration model. By matching in the preset obstacle model library according to the obstacle category, the corresponding three-dimensional obstacle restoration model is obtained to compensate the obstacle restoration model in the panoramic image.

[0055] S140: aligning the obstacle restoration model with the acquired obstacle coordinates and then superimposing them on the panoramic image.

[0056] In this embodiment, the obstacle coordinates are the coordinate values ​​of the obstacle obtained according to the real external environment, and the obstacle restoration model is aligned with the obstacle coordinates. Specifically, the real obstacle coordinates can be obtained according to the obstacle information. The obstacle restoration model itself is modeled based on a preset coordinate system, so the obstacle restoration model has an initial center coordinate. Therefore, it is necessary to align the obstacle restoration model in the coordinate system of the panoramic image according to the obstacle coordinates and superimpose it on the panoramic image. Specifically, the transparency of the obstacle restoration model can be adjusted to 50% and superimposed on the panoramic image on the vehicle's central control display screen. The specific superimposition effect can be determined by the attached Figure 7 As shown in the panoramic image compensation diagram, the 360 ​​video stream display area in the figure is the panoramic image displayed on the vehicle's central control display screen, the car is the obstacle, and the 3D obstacle restoration model is obtained after restoration processing through SR. The obstacle (car) coordinates are the coordinates in the panoramic image coordinate system with the car model as the coordinate origin, and the vehicle is the car model. The obstacle restoration model is aligned with the obtained obstacle coordinates and then superimposed on the panoramic image to compensate the panoramic image.

[0057] In one embodiment, if Figure 8 As shown, the step S140 also includes steps S141-S143.

[0058] S141, identifying the outline of the obstacle according to the acquired point cloud data corresponding to the obstacle information;

[0059] S142, performing center calculation on the contour to obtain obstacle coordinates;

[0060] S143: Align the obstacle restoration model with the obstacle coordinates and then superimpose them into the panoramic image.

[0061] In this embodiment, the obstacle coordinates are the coordinate values ​​of the obstacle obtained according to the real external environment. Specifically, the intelligent driving domain controller can identify the contours of the surrounding obstacles according to the point cloud data corresponding to the obstacle information, and calculate according to the contour information to obtain the obstacle coordinates, wherein the specific algorithm is not limited, and it is sufficient to obtain the obstacle coordinates. The central domain controller uses the car model as the center coordinate, aligns the obstacle restoration model with the obstacle coordinates obtained by its calculation, and then superimposes it in the panoramic image coordinate system to compensate the panoramic image. Specifically, the obstacle restoration model is superimposed on the panoramic image according to the obstacle coordinates. By superimposing the obstacle model on the panoramic image according to its corresponding coordinates, the panoramic image can be effectively compensated.

[0062] To further understand the panoramic image compensation method based on information fusion of the present invention, the following is an explanation through the data processing flow:

[0063] The vehicle body is equipped with a number of cameras, ultrasonic radars, and laser radars. In this embodiment, eleven cameras, twelve ultrasonic radars, and one laser radar are installed. The cameras, ultrasonic radars, and laser radars are connected to the ADCU (Autonomous Driving Control) through GMSL (Gigabit Multimedia Serial Links, a high-speed serial interface), CANFD (CAN with Flexible Data-Rate, an automotive communication protocol), and Eth (EtherNet, an automotive communication protocol based on Ethernet technology). Unit, intelligent driving domain controller), to transmit the collected environmental information and obstacle information to ADCU, ADCU senses the data sent by sensors and other devices, and fuses them to obtain the driving distance between the vehicle and the nearest obstacle, parking space information, obstacle coordinates, obstacle category, camera signal and other data, and in automatic parking mode, it can obtain the parking space to be parked according to the parking space information, and perform path planning to obtain the parking path. ADCU transmits the obstacle coordinates, obstacle category, camera signal and other data and parking path and other information to the central domain controller (CDCU) through Eth and GMSL. The central domain controller determines whether to trigger panoramic image compensation based on the received information. If triggered, it obtains the obstacle restoration model according to the obstacle category, and aligns it with the obstacle coordinates. The aligned obstacle restoration model is transmitted to the central control display screen through GMSL and superimposed with the panoramic image to compensate it. In addition, the method of the present invention can display the compensated panoramic image in both driving scenes and parking scenes, and the panoramic image can intuitively display environmental information to the user, which can effectively avoid users from frequently switching multiple APP pages.

[0064] Fig. 9 is a schematic block diagram of a panoramic image compensation device 200 based on information fusion provided by an embodiment of the present invention. Fig. 9 As shown, corresponding to the above panoramic image compensation method based on information fusion, the present invention also provides a panoramic image compensation device based on information fusion. The panoramic image compensation device based on information fusion includes a unit for executing the above panoramic image compensation method based on information fusion, and the device can be configured in a desktop computer, tablet computer, laptop computer, etc. For details, please refer to Fig. 9 The panoramic image compensation device based on information fusion includes a determination unit 210, a judgment unit 220, a restoration unit 230 and a superposition unit 240.

[0065] The determination unit 210 is used to determine the driving scene according to the driving speed and driving gear of the vehicle.

[0066] The judgment unit 220 is used to judge whether to trigger panoramic image compensation according to the collected environmental information and obstacle information and the driving scene.

[0067] In one embodiment, the determination unit 220 includes an acquisition unit and a first determination unit.

[0068] an acquisition unit, configured to acquire a driving distance between the vehicle and the obstacle closest to the vehicle if the current driving scene is a forward scene;

[0069] The first determination unit is used to determine whether the driving distance is less than a preset safety distance, and if so, determine to trigger panoramic image compensation.

[0070] In one embodiment, the determination unit 220 includes a parking determination unit, a feedback unit, and a second determination unit.

[0071] a parking determination unit, configured to determine a parking mode according to the acquired parking activation information if the current driving scene is a parking scene;

[0072] a feedback unit, configured to determine whether the panoramic image can feedback a clear image according to the light information and / or the rainfall information if the current parking mode is the manual parking mode;

[0073] The second determination unit is configured to determine whether to trigger panoramic image compensation if a clear image cannot be fed back.

[0074] In one embodiment, the determination unit 220 includes a detection unit and a third determination unit.

[0075] A detection unit, used to detect whether the camera signal used to feed back the panoramic image is normal;

[0076] The third determination unit is configured to determine whether to trigger panoramic image compensation if the camera signal fed back is abnormal.

[0077] In one embodiment, the determination unit 220 includes a selection unit, a stereo unit, a generation unit, and a fusion unit.

[0078] a selecting unit, configured to select a parking space to be parked according to the parking space information and obtain the four corner coordinates of the parking space to be parked if the current parking mode is the automatic parking mode, wherein the environmental information includes the parking space information;

[0079] A stereo unit, used for obtaining the parking area by using the three-dimensional restoration method to obtain the four-corner coordinates;

[0080] A generating unit, configured to generate a parking path by using a preset path planning method based on the current position information of the vehicle and the position of the parking space to be parked;

[0081] A fusion unit is configured to superimpose the parking area and the parking path into the panoramic image by using a preset fusion method.

[0082] The restoration unit 230 is used to restore the obstacle information by a three-dimensional restoration method to obtain an obstacle restoration model if the panoramic image compensation is triggered.

[0083] In one embodiment, the sending unit 230 includes a category identification unit and a matching unit.

[0084] A category recognition unit, used to obtain point cloud data corresponding to the obstacle information, and perform type recognition according to the point cloud data to obtain the obstacle category;

[0085] The matching unit is used to perform type matching in a preset obstacle model library according to the obstacle category to obtain the obstacle restoration model.

[0086] The superimposition unit 240 is used to align the obstacle restoration model with the acquired obstacle coordinates and then superimpose them on the panoramic image.

[0087] In one embodiment, the superposition unit 240 includes a contour recognition unit, a calculation unit, and an alignment unit.

[0088] A contour recognition unit, used to recognize the contour of the obstacle based on the acquired point cloud data corresponding to the obstacle information;

[0089] A calculation unit, used for performing center calculation on the contour to obtain the coordinates of the obstacle;

[0090] An alignment unit is used to align the obstacle restoration model with the obstacle coordinates and then superimpose them into the panoramic image.

[0091] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the above-mentioned panoramic image compensation device 200 based on information fusion and each unit can refer to the corresponding description in the aforementioned method embodiment, and for the convenience and brevity of description, it will not be repeated here.

[0092] The above-mentioned panoramic image compensation device based on information fusion can be implemented in the form of a computer program. The computer program can be used in Fig.10 Runs on the computer device shown.

[0093] See also Fig.10 , Fig.105 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device 500 may be a terminal or a server, wherein the terminal may be an electronic device with communication functions such as a smart phone, a tablet computer, a laptop computer, a desktop computer, a personal digital assistant, and a wearable device. The server may be an independent server or a server cluster composed of multiple servers.

[0094] See also Fig.10 The computer device 500 includes a processor 502 , a memory and a network interface 505 connected via a system bus 501 , wherein the memory may include a non-volatile storage medium 503 and an internal memory 504 .

[0095] The non-volatile storage medium 503 can store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions, and when the program instructions are executed, the processor 502 can execute a panoramic image compensation method based on information fusion.

[0096] The processor 502 is used to provide computing and control capabilities to support the operation of the entire computer device 500 .

[0097] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a panoramic image compensation method based on information fusion.

[0098] The network interface 505 is used to communicate with other devices over the network. Fig.10 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present application, and does not constitute a limitation on the computer device 500 to which the solution of the present application is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0099] The processor 502 is used to run a computer program 5032 stored in the memory to implement the steps of the above method.

[0100] It should be understood that in the embodiment of the present application, the processor 502 may be a central processing unit (CPU), and the processor 502 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0101] It can be understood by those skilled in the art that all or part of the processes in the method for implementing the above embodiment can be completed by instructing the relevant hardware through a computer program. The computer program includes program instructions, and the computer program can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiment of the above method.

[0102] Therefore, the present invention also provides a storage medium. The storage medium may be a computer-readable storage medium. The storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a processor, the processor executes the steps of the above method.

[0103] The storage medium may be a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk, etc., which are computer-readable storage media that can store program codes.

[0104] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0105] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of each unit is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0106] The steps in the method of the embodiment of the present invention can be adjusted in order, combined and deleted according to actual needs. The units in the device of the embodiment of the present invention can be combined, divided and deleted according to actual needs. In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0107] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, terminal, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention.

[0108] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A panoramic image compensation method based on information fusion, characterized in that: The method comprises: Determine the driving scenario according to the vehicle's driving speed and driving gear; Determining whether to trigger panoramic image compensation based on the collected environmental information and obstacle information and the driving scene; If the panoramic image compensation is triggered, the obstacle information is restored by a three-dimensional restoration method to obtain an obstacle restoration model; The obstacle restoration model is aligned with the acquired obstacle coordinates and then superimposed on the panoramic image.

2. The method according to claim 1, characterized in that The driving scene includes a forward scene, and the step of determining whether to trigger panoramic image compensation according to the collected environment information and obstacle information and the driving scene includes: If the current driving scene is a forward scene, obtaining the driving distance between the vehicle and the obstacle closest to it; It is determined whether the driving distance is less than a preset safety distance, and if so, panoramic image compensation is triggered.

3. The method according to claim 1, characterized in that The driving scene includes a parking scene, the environmental information includes light information and / or rainfall information, and the step of determining whether to trigger panoramic image compensation based on the collected environmental information and obstacle information and the driving scene includes: If the current driving scene is a parking scene, determining the parking mode according to the acquired parking activation information; If the current parking mode is the manual parking mode, determining whether the panoramic image can feedback a clear image according to the light information and / or the rainfall information; If a clear image cannot be fed back, it is determined that panoramic image compensation is triggered.

4. The method according to any one of claims 1 to 3, characterized in that The step of determining whether to trigger panoramic image compensation based on the collected environment information and obstacle information and the driving scene also includes: Detecting whether the camera signal used to feed back the panoramic image is normal; If the camera signal fed back is abnormal, it is determined that panoramic image compensation is triggered.

5. The method according to claim 3, characterized in that: The step of determining the parking mode according to the acquired parking activation information further includes: If the current parking mode is the automatic parking mode, a parking space to be parked is selected according to the parking space information, and the four corner coordinates of the parking space to be parked are obtained, wherein the environmental information includes the parking space information; The four corner coordinates are obtained by the three-dimensional restoration method to obtain the parking area; Generate a parking path by using a preset path planning method based on the current position information of the vehicle and the position of the parking space to be parked; The parking area and the parking path are superimposed on the panoramic image through a preset fusion method.

6. The method according to any one of claims 1 to 3, characterized in that: The step of restoring the obstacle information by a three-dimensional restoration method to obtain an obstacle restoration model includes: Acquire point cloud data corresponding to the obstacle information, and perform type recognition according to the point cloud data to acquire the obstacle category; According to the obstacle category, type matching is performed in a preset obstacle model library to obtain the obstacle restoration model.

7. The method according to claim 6, characterized in that The step of aligning the obstacle restoration model with the acquired obstacle coordinates and then superimposing them on the panoramic image comprises: Identifying the outline of the obstacle based on the acquired point cloud data corresponding to the obstacle information; Calculate the center of the contour to obtain the coordinates of the obstacle; The obstacle restoration model is aligned with the obstacle coordinates and then superimposed on the panoramic image.

8. A panoramic image compensation device based on information fusion, characterized in that: include: A determination unit, used to determine a driving scene according to a driving speed and a driving gear of the vehicle; A judgment unit, used to judge whether to trigger panoramic image compensation according to the collected environmental information and obstacle information and the driving scene; A restoration unit, configured to restore the obstacle information by a three-dimensional restoration method to obtain an obstacle restoration model if panoramic image compensation is triggered; The superimposition unit is used to align the obstacle restoration model with the acquired obstacle coordinates and then superimpose them on the panoramic image.

9. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the method according to any one of claims 1 to 7 can be implemented.

Images