Image processing method and device, electronic equipment and vehicle

Abstract

The invention provides an image processing method and device, electronic equipment and a vehicle, and the method comprises the steps: collecting an image of an environment where the current vehicle is located, and obtaining a real environment image; identifying the types of environmental elements in the real environment image to obtain at least one type label; respectively acquiring a scene image corresponding to each type label from an external database to obtain a plurality of scene element images; and constructing and obtaining a three-dimensional panoramic image simulating the surrounding environment of the current vehicle based on the plurality of scene element images. According to the image processing method and device, the electronic equipment and the vehicle disclosed by the invention, the three-dimensional panoramic image simulating the surrounding environment of the current vehicle can be constructed, a driver is prevented from parking and getting off to observe the environment, and the driving safety and the driving experience are effectively improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to image processing methods, devices, electronic equipment and vehicles. Background Art

[0002] When the vehicle is driving, the driver is in the vehicle with a limited field of vision. Sometimes, in order to better observe the surrounding environment, the driver has to get off the vehicle to observe. However, the timing of stopping and getting off the vehicle on the road is limited, and it is impossible to stop and check the environment at any time, which affects driving safety and driving experience. Summary of the invention

[0003] In view of this, the purpose of this application is to propose an image processing method, device, electronic device and vehicle to improve driving experience and driving safety.

[0004] Based on the above purpose, the present application provides an image processing method, comprising:

[0005] Collect the image of the environment where the current vehicle is located to obtain the real environment image;

[0006] Identify the type of environmental elements in the real environment image to obtain at least one type label;

[0007] Acquire scene images corresponding to each of the type labels from an external database to obtain multiple scene element images;

[0008] A three-dimensional panoramic image simulating the current vehicle surrounding environment is constructed based on multiple scene element images.

[0009] Optionally, constructing a three-dimensional panoramic image simulating the surrounding environment of the current vehicle based on multiple scene element images includes:

[0010] Using image stitching technology to stitch the multiple scene element images into a planar panoramic image;

[0011] The planar panoramic image is mapped into a spherical panoramic image by using a three-dimensional engine technology to obtain the three-dimensional panoramic image.

[0012] Optionally, before mapping the planar panoramic image into a spherical panoramic image using a three-dimensional engine technology, the method further includes:

[0013] The image object removal technology is used to remove objects in the planar panoramic image that are below a preset height threshold from the ground.

[0014] Optionally, it also includes:

[0015] Obtain the three-dimensional model data of the current vehicle;

[0016] The three-dimensional model data of the current vehicle is fused with the three-dimensional panoramic image to obtain a three-dimensional panoramic image including the three-dimensional model of the current vehicle.

[0017] Optionally, the real environment image includes a vehicle forward environment image, and the method further includes:

[0018] In response to the type label corresponding to the vehicle forward environment image being in compliance with a preset label range that affects driving safety, a prompt message is generated and issued.

[0019] Optionally, generating and issuing prompt information includes:

[0020] Corresponding warning information and / or driving advice is generated according to the type label, and the warning information and / or driving advice is fused with the three-dimensional panoramic image to obtain a three-dimensional panoramic image containing forward environment prompt information.

[0021] Optionally, it also includes:

[0022] Get the navigation information of the current vehicle;

[0023] Generate driving direction guidance information based on the navigation information and the real environment image;

[0024] The driving direction guidance information is fused with the three-dimensional panoramic image to obtain a three-dimensional panoramic image containing the driving direction guidance.

[0025] The second aspect of the present application also provides an image processing device, comprising:

[0026] The acquisition module is used to acquire the image of the environment in which the vehicle is currently located to obtain the real environment image;

[0027] An element recognition module, used to recognize the type of environmental elements in the real environment image and obtain at least one type label;

[0028] An image acquisition module, used to acquire scene images corresponding to each type label from an external database to obtain multiple scene element images;

[0029] The generation module is used to construct a three-dimensional panoramic image simulating the current vehicle surrounding environment based on multiple scene element images.

[0030] The third aspect of the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable by the processor, wherein the processor implements the method as described above when executing the computer program.

[0031] A fourth aspect of the present application also provides a vehicle, which includes the electronic device as described above.

[0032] From the above, it can be seen that the image processing method, device, electronic device and vehicle provided by the present application first collect and obtain the real environment image, then identify the type of environmental elements in the real environment image, and obtain the corresponding type label. At this time, the type label can represent the real environment state, so the scene element image obtained from the external database using the type label can accurately represent the real environment situation; finally, the three-dimensional panoramic image constructed based on multiple scene element images can effectively simulate the current vehicle surrounding environment. During the driving process of the vehicle, the real image of the current vehicle environment can be observed through the three-dimensional panoramic image, without stopping or getting off the vehicle to observe, and the driving safety is high and the driving experience is good. At the same time, the scene image is expanded by using type labels and external databases, and there is no need to collect a large number of real environment images for analysis, and the processing efficiency and calculation efficiency are high; and the scene element images are stored in the external database, which has little restriction on the database storage space, and can store as many different scene images as possible, with a wider range of applications and higher calculation efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the present application or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0034] Figure 1 A schematic diagram of the flow of an image processing method according to an embodiment of the present application;

[0035] Figure 2 A schematic diagram of the structure of an image processing device according to an embodiment of the present application;

[0036] Figure 3 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0037] In order to make the objectives, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0038] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present application should be the usual meanings understood by people with ordinary skills in the field to which the present application belongs. The "first", "second" and similar words used in the embodiments of the present application do not represent any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing in front of the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0039] As described in the background technology, in order to ensure driving safety, the driver needs to constantly observe the surrounding environment of the vehicle and constantly adjust the driving parameters according to the surrounding environment of the vehicle during the driving process. For example, when there is water accumulation or ice on the road surface, the speed needs to be reduced; when the road is sandy, the driving gear needs to be reduced; when the surrounding environment is a city with dense traffic, the speed needs to be reduced, etc. Therefore, the situation of the surrounding environment of the vehicle is closely related to the driving parameters and driving safety. However, when the driver sits in the car, the field of vision is limited, the range that can be observed is limited, and the accuracy of observation is also poor. When the driver needs to accurately observe the surrounding environment of the vehicle, he has to stop and get off to observe, which not only affects driving safety, but also reduces the driving experience. In addition, with the development of intelligent driving technology, people's requirements for driving experience are gradually increasing, and technologies such as smart cockpits and smart driving are also increasingly being used. However, in the current smart cockpits and smart driving technologies, drivers and passengers cannot accurately understand the external real environment, and the driving experience is not good.

[0040] In view of this, the present application provides an image processing method that can construct a three-dimensional panoramic image that simulates the current vehicle surrounding environment, so that the driver and passengers can observe the real situation outside the vehicle without getting off the vehicle, with high driving safety and good driving experience. Figure 1 As shown, the method includes:

[0041] S101, collecting an image of the environment in which the vehicle is currently located to obtain a real environment image;

[0042] Specifically, the camera device (such as a driving recorder, etc.) on the vehicle can be used to collect images of the environment in which the current vehicle is located. When the camera device is used to collect images of the environment in which the current vehicle is located, the camera device can be used to shoot real-time video of the external environment of the vehicle, and then the image frame can be intercepted from the video stream as the real environment image, or the camera device can be used to take photos of the external environment of the vehicle in real time to obtain the real environment image of the external environment, without specific restrictions. The camera device can be installed inside the vehicle or outside the vehicle, without specific restrictions. Under normal circumstances, the sight line in front of the vehicle is relatively open, and the range that can be observed is relatively wide. The content observed through the front field of view can basically simulate the panoramic range image of the vehicle; and as the vehicle travels, the forward image at the previous moment will become the backward image at the next moment. Therefore, only collecting the real environment image in the front of the vehicle can also simulate the panoramic image of the vehicle. Therefore, the camera device used to collect images of the environment in which the current vehicle is located in this application can be an ordinary camera device, that is, it can only collect environmental images in the direction of the vehicle's forward movement. Of course, in order to collect more real environment images in multiple directions, the camera equipment can also use panoramic camera equipment, such as panoramic cameras, panoramic driving recorders, etc., which can collect 360° panoramic images around the vehicle and obtain real environment images in multiple directions, providing a prerequisite for the subsequent construction of accurate three-dimensional panoramic images, without specific restrictions.

[0043] S102, identifying the type of environmental elements in the real environment image, and obtaining at least one type label;

[0044] Specifically, the environmental factors include at least one of the driving road surface, road surface conditions, weather, surrounding environment, traffic conditions, surrounding buildings, and surrounding objects. The type of driving road surface refers to the type of road on which the current vehicle is driving, including: highway, expressway, mud, sand, rock road, gravel road, grassland, etc. The type of road surface conditions refers to the specific conditions of the road on which the current vehicle is driving, including ice, potholes, snow, water, etc. The type of weather refers to the real weather of the current vehicle's environment, including: light rain, heavy rain, snow, heavy snow, fog, hail, sunny days, etc. The type of surrounding environment refers to the surrounding environment of the current vehicle, including: rural, urban, desert, grassland, sea, forest, etc. The type of traffic conditions refers to the traffic conditions of the road on which the current vehicle is driving, including: unobstructed, basically unobstructed, light congestion, moderate congestion, severe congestion, etc. The type of surrounding buildings refers to the type of buildings around the current vehicle, including: residential buildings, business buildings, tile houses, shopping malls, etc. The type of surrounding objects refers to the specific types of objects around the current vehicle, such as cars, motorcycles, bicycles, tricycles, pedestrians, walls, etc. The type label of weather can further include visibility information, the type label of surrounding buildings can also include building dimensions, such as total height, floor height, width, etc.; the type label of road conditions can also include road flatness information; the type label of surrounding objects can also include vehicle model information, that is, when the surrounding object is identified as a car, the vehicle model information is further identified, such as sports cars, off-road vehicles, etc., and the specific car brand and precise model can also be further identified, thereby obtaining more dimensional real environment information. According to the actual application situation, the content included in the environmental factors can be further broadened, the type of each environmental factor can be further expanded, or the type of environmental factors can be divided into different dimensions according to different application situations, without specific restrictions.

[0045] When the real environment image is obtained, the type of environmental elements in the real environment image can be identified through the image recognition algorithm. For example, the current real environment image is an image of a city road on a rainy day. The image recognition algorithm can be used to identify that the corresponding type labels may include: city, highway, light rain, and accumulated water. This can be used to analyze the environmental elements of the vehicle's current environment, providing a basis for subsequently obtaining accurate scene element images.

[0046] The image recognition algorithm can be a convolutional neural network (CNN), a generative adversarial network (GAN), a regional convolutional neural network (R-CNN), YOLO, a visual Transformer and other algorithms. Other algorithms that can realize image recognition can also be applied to this application without specific limitation.

[0047] S103, respectively acquiring scene images corresponding to each of the type labels from an external database to obtain a plurality of scene element images;

[0048] Specifically, after obtaining the type label, the type label is used to obtain the scene images corresponding to each type label from the external database. For example, when the type label is city, highway, light rain, and water accumulation, the scene images corresponding to city, highway, light rain, and water accumulation are obtained from the external database, respectively, so as to obtain multiple scene element images. When constructing the external database, the types of environmental elements of all images in the external database are first identified, so that each image can correspond to at least one preset label; then, according to the preset labels, the environmental elements are grouped and classified, that is, the driving road surface, road conditions, weather, surrounding environment, traffic conditions, surrounding buildings, and surrounding objects are first divided into groups, and then all the images in the external database are classified according to the above groups. When it is necessary to obtain multiple scene element images from the external database, first, according to the environmental elements corresponding to the type label, the image under the corresponding environmental element group in the external database is retrieved, and then the type label is compared with the preset label of the image under the corresponding environmental element group, so as to obtain the corresponding scene element image. Since the images in the external database are grouped and classified according to the environmental elements, it is not necessary to compare the labels of all images in the database, which can effectively improve the efficiency of obtaining scene element images.

[0049] The external database can be a traditional relational database, such as MySQL, PostgreSQL, or a non-relational database (NoSQL), such as MongoDB, Cassandra, or a distributed file system (HDFS), etc., without specific restrictions. The external database can be installed on a cloud server, and after obtaining the type label, it is transmitted to the cloud server by wireless or other means, and then the corresponding scene element image is obtained from the external database of the cloud server. The external database can also be deployed in a hybrid cloud architecture and other ways, connecting the external database through a cloud server, that is, after obtaining the type label, the type label is transmitted to the cloud server, and then the cloud server sends instructions to the specific external data to retrieve the scene element image, without specific restrictions.

[0050] S104: constructing a three-dimensional panoramic image simulating the surrounding environment of the current vehicle based on the multiple scene element images.

[0051] Specifically, after obtaining multiple scene element images through type tags, multiple scene element images are used to simulate the surrounding environment of the current vehicle, thereby obtaining a three-dimensional panoramic image. The scene image is obtained based on the environmental elements in the real environment image, so that the final simulated three-dimensional panoramic image can accurately restore the real environment of the current vehicle. Drivers and passengers can understand the real environment outside the vehicle through the three-dimensional panoramic image, with a wider field of view, more accurate information, no need to stop and get off the vehicle, and better driving safety and experience.

[0052] In this application, based on the steps of S101 to S104, firstly, the real environment image is acquired, and then the type of environmental elements in the real environment image is identified to obtain the corresponding type label. At this time, the type label can represent the real environment state, so the scene element image obtained from the external database using the type label can accurately represent the real environment; finally, the three-dimensional panoramic image constructed based on multiple scene element images can effectively simulate the current vehicle surrounding environment. During the driving process of the vehicle, the real image of the current vehicle environment can be observed through the three-dimensional panoramic image, without stopping or getting off the vehicle to observe, and the driving safety is high and the driving experience is good. At the same time, the scene image is expanded by using the type label and the external database, and there is no need to collect a large number of real environment images for analysis, and the processing efficiency and calculation efficiency are high; and the scene element image is stored in the external database, which has little restriction on the database storage space, and can store as many different scene images as possible, with a wider range of applications and higher calculation efficiency.

[0053] The image processing method of the present application can be applied to technologies such as smart cockpits or smart driving. Smart cockpit refers to a car interior environment created by using advanced information technology, artificial intelligence, the Internet of Things, big data and other technologies, which can enhance the driving experience, convenience and safety of car owners. The smart cockpit not only includes traditional hardware equipment such as dashboards and seats, but also integrates a variety of intelligent systems and services. Intelligent driving technology refers to the use of artificial intelligence, machine learning, computer vision, laser radar, cameras, radar and other technologies to enable cars to autonomously perceive, understand, make decisions and control, thereby realizing unmanned driving or assisted driving functions. Whether it is a smart cockpit or smart driving technology, a control system, i.e., a car system, needs to be loaded on the vehicle. In the car system, a central control screen is usually provided, and human-computer interaction can be achieved through the central control screen. When the present application is applied to a smart cockpit or smart driving, a three-dimensional panoramic image can be displayed on the central control screen in the vehicle, and interaction with the user can be achieved through the central control screen. During the driving process of the vehicle, real-world environment images can be collected in real time, and the types of environmental elements in the real-world environment images can be identified in real time to obtain type labels. Then, multiple scene element images can be obtained from an external database in real time according to the type labels, and a three-dimensional panoramic image can be constructed in real time. Finally, the real-time generated three-dimensional panoramic image can be displayed on the central control screen, so that the three-dimensional panoramic image can change in real time according to the driving conditions of the vehicle, and the external environment of the vehicle can be displayed in the vehicle in time. The driver and passengers can observe the surrounding environment of the vehicle without getting off the vehicle, and the driving efficiency and safety are higher, thereby improving the driving experience and driving safety of the smart cockpit and smart driving. At the same time, an external database is used to store and obtain scene element images. The vehicle system does not need to deploy an external database separately, which will not occupy the vehicle system resources and has higher computing efficiency.

[0054] The construction of the three-dimensional panoramic image can be achieved by using image stitching technology and three-dimensional engine technology. In some embodiments, the three-dimensional panoramic image simulating the surrounding environment of the current vehicle based on multiple scene element images includes:

[0055] Using image stitching technology to stitch the multiple scene element images into a planar panoramic image;

[0056] The planar panoramic image is mapped into a spherical panoramic image by using a three-dimensional engine technology to obtain the three-dimensional panoramic image.

[0057] Specifically, image stitching technology is a technology that combines multiple images into a wide or high-resolution image, and is often used in the production of panoramic images. The image stitching technology of the present application can be based on a spatial domain image stitching algorithm, a stitching algorithm based on normalized cross-correlation, an image stitching algorithm based on mutual information, a stitching algorithm based on underlying features, etc. Other technologies that can achieve image stitching can also be applied to the present application, without specific limitation. Exemplarily, the image stitching technology may include image preprocessing, feature detection and matching, image registration, image transformation and stitching, and image fusion and optimization steps. The specific execution content of each step is as follows: (1) Image preprocessing: perform preliminary processing on the input multiple scene element images, such as denoising, grayscale, contrast adjustment, etc., to improve the stability of subsequent processing; (2) Feature detection and matching: use the feature point detection algorithm to find key feature points (such as corner points or texture points) in each image, and use the matching algorithm to find the matching point pairs between different scene element images; (3) Image registration: based on the matched feature points, calculate the transformation matrix (perspective transformation matrix) from one scene element image to another scene element image to ensure the scene The geometric relationship between the element images is accurately aligned; (4) Image transformation and stitching: Using the calculated transformation matrix, each scene element image is mapped to a common plane according to its transformation relationship. This step usually requires geometric transformation of the scene element images, such as rotation, scaling, translation, etc., so that different scene element images can be seamlessly connected, thereby obtaining the initial planar panoramic image; (5) Image fusion and optimization: The stitched image may have problems such as seams and color aberration. Therefore, image fusion technology, such as multi-resolution fusion, gradient stitching or optimized smooth seams, can be used to process the initial planar panoramic image to reduce the visibility of the seams and improve the quality of the final composite image, thereby obtaining the final planar panoramic image. Of course, other steps can also be used to process multiple scene element images to obtain a planar panoramic image, and there is no specific limitation.

[0058] Three-dimensional engine technology (3D engine technology) is a computer graphics tool and software framework for creating and rendering three-dimensional graphics, scene interactions and virtual environments, which can provide functions for processing and displaying three-dimensional models, scenes, lighting, textures, animations and other elements. In three-dimensional engine technology, skybox technology is usually used to simulate a large range of environments, such as the sky, distant horizons, clouds, stars, etc. The skybox allows observers to perceive the vastness of the virtual world in a large and endless environment. In this application, the skybox technology in the three-dimensional engine technology can be used to map a planar panoramic image into a spherical panoramic image to obtain a three-dimensional panoramic image. The three-dimensional panoramic image is obtained by determining the scene image according to the environmental elements in the real environment, which can effectively reflect the current situation of the real environment of the vehicle; at the same time, the three-dimensional panoramic image is a spherical panoramic image, and the user has a full range of vision, which is convenient for the user to observe the full range of information around the vehicle. The user does not need to stop the vehicle to observe the environment around the vehicle in real time, and the driving efficiency, driving safety and driving experience are better.

[0059] After obtaining the planar panoramic image, there may be some debris or objects that affect the three-dimensional effect in the planar panoramic image, and some technology can be used to remove them. In some embodiments, before mapping the planar panoramic image into a spherical panoramic image using the three-dimensional engine technology, it also includes:

[0060] The image object removal technology is used to remove objects in the planar panoramic image that are below a preset height threshold from the ground.

[0061] Specifically, image object removal technology can automatically or semi-automatically remove certain objects or elements from an image while maintaining the coherence and naturalness of other parts of the image. It is often used in scenes such as image editing, visual restoration, and landscaping. In the 3D reconstruction process, parallax refers to the change in the position of an object in an image as the viewing angle changes. The smaller the size of the object, the less obvious the parallax effect. When the objects are very small (for example, less than 10 cm), their displacement under multiple perspectives can hardly be captured, which makes it impossible to accurately infer the depth information of these objects during 3D reconstruction. At the same time, small objects often introduce more image noise, especially in low-resolution images or when the shooting environment is not well lit. These noises may interfere with the effect of the 3D reconstruction algorithm, resulting in the inability to effectively remove or repair small objects in the reconstruction results, affecting the final 3D image quality. The goal of a 3D panorama is to generate a panoramic effect that can be freely browsed in a virtual space and has visual consistency. When these small objects exist, unnatural seams or shape distortions may occur during the stitching process, especially between overlapping areas of different images. In summary, when there are some small objects in the planar panoramic image, the three-dimensional construction will cause problems such as unclear parallax, inaccurate depth reconstruction, and noise interference. These small objects are difficult to provide sufficient depth information between multiple perspectives, which will eventually affect the three-dimensional reconstruction effect, thereby causing the quality of the three-dimensional panorama to deteriorate. Therefore, before using the planar panoramic image to construct a three-dimensional panoramic image, the image removal technology is used to remove objects below the preset height threshold from the ground in the planar panoramic image, and then the three-dimensional panoramic image is constructed. This can effectively improve the accuracy and visual consistency, ensure the smooth transition of the panorama under different perspectives, and at the same time, small objects are removed, which reduces the amount of calculation and effectively improves the calculation efficiency. Optionally, the preset height threshold is 10cm, and can also be 15, 14, 13, 12, 11, 9, 8, 7, 6, 5, 4, 3, 2, 1cm, or it can be set to other values ​​according to actual conditions, without specific restrictions.

[0062] Integrating the 3D model of the vehicle into the 3D panoramic image can further improve the driving experience. In some embodiments, the method further includes:

[0063] Obtain the three-dimensional model data of the current vehicle;

[0064] The three-dimensional model data of the current vehicle is fused with the three-dimensional panoramic image to obtain a three-dimensional panoramic image including the three-dimensional model of the current vehicle.

[0065] Specifically, the three-dimensional model data of the current vehicle is fused with the three-dimensional panoramic image to obtain a three-dimensional panoramic image including the three-dimensional model of the current vehicle. Users can observe the environment around the vehicle more realistically, accurately and intuitively, thereby effectively improving driving experience and driving safety.

[0066] After the type label is obtained, a prompt message can be generated and issued according to the type label to remind the driver to pay attention. In some embodiments, the real environment image includes a vehicle front environment image, and the method further includes:

[0067] In response to the type label corresponding to the vehicle forward environment image being in compliance with a preset label range that affects driving safety, a prompt message is generated and issued.

[0068] Specifically, a preset tag range is set in advance, and the type tags corresponding to the situations that affect driving safety are classified therein. After the type tag is identified by the real environment image, it is determined whether the type tag is within the preset tag range. If so, it means that there is a situation that affects driving safety in the current environment. At this time, a prompt message is generated and issued according to the content of the type tag. Exemplarily, if the identified type tag includes "ice", the preset tag range also includes "ice", and the icy road is likely to cause the vehicle to slip, which directly affects driving safety. At this time, a prompt message is generated and issued according to the type tag of "ice", such as "the road ahead is icy", "slow down driving", etc., to remind the driver to pay attention, thereby improving driving safety. Exemplarily, if the identified type tag includes "water accumulation", the preset tag range also includes "water accumulation", and the waterlogged road may cause the vehicle to stall. At this time, a prompt message is generated and issued according to the type tag of "water accumulation", such as "water accumulation ahead", "constant speed driving", etc. During the vehicle driving process, prompt information is generated and issued in real time, which plays a role in predicting the driving direction. The user can change and adjust the driving parameters in time according to the prompt information, thereby further improving driving safety.

[0069] The prompt information may be further integrated into the three-dimensional panoramic image. In some embodiments, the generating and issuing of the prompt information includes:

[0070] Corresponding warning information and / or driving advice is generated according to the type label, and the warning information and / or driving advice is fused with the three-dimensional panoramic image to obtain a three-dimensional panoramic image containing forward environment prompt information.

[0071] Specifically, the prompt information may include warning information and driving advice. When there is a situation that affects driving safety in the environment where the vehicle is located, the corresponding warning information or driving advice is generated according to the type label, and the warning information and driving advice can also be generated at the same time, and then the warning information and driving advice are fused with the three-dimensional panoramic image and displayed in the three-dimensional panoramic image, thereby reminding the driver to drive carefully and improve driving safety. Exemplarily, when the type label is "icing", corresponding warning information such as "the road ahead is icy" can be generated, and corresponding driving advice such as "low gear and low speed forward" can also be generated, and the information of "the road ahead is icy" and "low gear and low speed forward" is fused with the three-dimensional panoramic image, so that "the road ahead is icy" and "low gear and low speed forward" can be displayed in the three-dimensional panoramic image, and the user can observe the surrounding situation in time through the three-dimensional panoramic image, and adjust the driving parameters in time according to the warning information and driving advice, thereby improving driving safety. Optionally, warning information and driving suggestions can be displayed in the three-dimensional panoramic image at the position of the content corresponding to the type label. Taking the type label "ice" as an example, the corresponding information such as "the road ahead is icy" and "drive at low gear and low speed" can be directly displayed at the location where there is ice in the three-dimensional panoramic image, so as to better remind the driver to avoid or drive, and better improve driving safety.

[0072] The navigation information may be further integrated into the three-dimensional panoramic image. In some embodiments, the navigation information of the current vehicle is obtained;

[0073] Generate driving direction guidance information based on the navigation information and the real environment image;

[0074] The driving direction guidance information is fused with the three-dimensional panoramic image to obtain a three-dimensional panoramic image containing the driving direction guidance.

[0075] Specifically, driving direction guidance information is generated according to navigation information, and then it is merged with the three-dimensional panoramic image, so that the driving direction guidance information is displayed in the three-dimensional panoramic image, which can better guide users to drive and improve driving safety and driving experience. For example, when driving to a fork in the road, driving direction guidance information such as guidance arrows are generated according to the navigation information; when changing lanes or overtaking is required, driving direction guidance information such as lane change path and overtaking path are generated according to the navigation information, and then the driving direction guidance information such as lane change path and overtaking path are merged with the three-dimensional panoramic image. The driver can intuitively determine the driving direction related information such as the lane change path or overtaking path by observing the three-dimensional panoramic image, which has higher driving efficiency and better driving experience.

[0076] When applied to smart cockpits or smart driving, the three-dimensional panoramic image is displayed on the central control screen, and warning information, driving suggestions, driving direction guidance information, etc. are displayed and updated together with the three-dimensional panoramic image on the central control screen, making user observation more intuitive and the driving experience better.

[0077] The image processing method provided by the present application obtains a real environment image by acquisition, and then identifies the type of environmental elements in the real environment image to obtain a corresponding type label. At this time, the type label can represent the real environment state, so that the scene element image obtained from the external database using the type label can accurately represent the situation of the real environment; and the external database does not need to be installed in the vehicle system, which can effectively reduce the storage limit of the vehicle system and improve the computing efficiency; and then a plane panoramic image is obtained by splicing multiple scene element images, and finally the plane panoramic image is mapped into a spherical panoramic image using the three-dimensional engine technology, which can effectively reflect the situation of the real environment in which the current vehicle is located; at the same time, the three-dimensional panoramic image is a spherical panoramic image, and the user has a full range of vision, which is convenient for the user to observe the full range of information around the vehicle. The user does not need to stop or get off the vehicle to observe the environment around the vehicle in real time, and the driving efficiency, driving safety, and driving experience are better. On the basis of the above, the three-dimensional model of the vehicle, warning information, driving suggestions, driving direction guidance information, etc. can also be integrated into the three-dimensional panoramic image, so that the driver can obtain information more intuitively, and then the driver can be better guided to drive, further improving driving safety and driving experience.

[0078] It should be noted that the method of the embodiment of the present application can be performed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present application, and the multiple devices will interact with each other to complete the described method.

[0079] It should be noted that the above describes some embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0080] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present application also provides an image processing device, referring to Figure 2 , the image processing device comprises:

[0081] The acquisition module 201 is used to acquire an image of the environment in which the vehicle is currently located to obtain a real environment image;

[0082] An element recognition module 202 is used to recognize the type of environmental elements in the real environment image and obtain at least one type label;

[0083] An image acquisition module 203 is used to acquire scene images corresponding to each type label from an external database to obtain multiple scene element images;

[0084] The generating module 204 is used to construct a three-dimensional panoramic image simulating the surrounding environment of the current vehicle based on multiple scene element images.

[0085] In some embodiments, the generation module 204 is further configured to:

[0086] Using image stitching technology to stitch the multiple scene element images into a planar panoramic image;

[0087] The planar panoramic image is mapped into a spherical panoramic image by using a three-dimensional engine technology to obtain the three-dimensional panoramic image.

[0088] In some embodiments, before mapping the planar panoramic image into a spherical panoramic image using a three-dimensional engine technology, the method further includes:

[0089] The image object removal technology is used to remove objects in the planar panoramic image that are below a preset height threshold from the ground.

[0090] In some embodiments, it also includes:

[0091] A first acquisition module is used to acquire three-dimensional model data of the current vehicle;

[0092] The first fusion module is used to fuse the three-dimensional model data of the current vehicle with the three-dimensional panoramic image to obtain a three-dimensional panoramic image including the three-dimensional model of the current vehicle.

[0093] In some embodiments, the real environment image includes a vehicle front environment image, and the device further includes:

[0094] Reminder module: generates and issues a reminder message in response to the type label corresponding to the vehicle's forward environment image being in accordance with a preset label range that affects driving safety.

[0095] In some implementations, generating and issuing prompt information includes:

[0096] Corresponding warning information and / or driving advice is generated according to the type label, and the warning information and / or driving advice is fused with the three-dimensional panoramic image to obtain a three-dimensional panoramic image containing forward environment prompt information.

[0097] In some embodiments, it also includes:

[0098] The second acquisition module is used to acquire navigation information of the current vehicle;

[0099] An information generation module, used for generating driving direction guidance information based on the navigation information and the real environment image;

[0100] The second fusion module is used to fuse the driving direction guidance information with the three-dimensional panoramic image to obtain a three-dimensional panoramic image containing the driving direction guidance.

[0101] For the convenience of description, the above device is described in terms of functions divided into various modules. Of course, when implementing the present application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0102] The device of the above embodiment is used to implement the corresponding image processing method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0103] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the image processing method described in any of the above embodiments is implemented.

[0104] Figure 3 A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 in the device.

[0105] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0106] The memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 may store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1020 and are called and executed by the processor 1010.

[0107] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0108] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0109] The bus 1050 includes a path that transmits information between the various components of the device (eg, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040).

[0110] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0111] The electronic device of the above embodiment is used to implement the corresponding image processing method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0112] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the image processing method described in any of the above embodiments.

[0113] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0114] The computer instructions stored in the storage medium of the above embodiments are used to enable the computer to execute the image processing method described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0115] Based on the same concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a computer program product, including computer program instructions. When the computer program instructions are run on a computer, the computer executes the image processing method described in any of the above embodiments, which has the beneficial effects of the corresponding method embodiments and will not be repeated here.

[0116] It is understandable that before using the technical solutions of each embodiment of the present disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved will be informed to the user in an appropriate manner, and the user's authorization will be obtained.

[0117] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly remind the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can independently choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0118] As an optional but non-limiting implementation, in response to receiving the user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0119] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0120] A person skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application is limited to these examples. In line with the concept of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0121] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present application difficult to understand, the known power supply / ground connection with the integrated circuit (IC) chip and other components may or may not be shown in the provided drawings. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform to be implemented in the embodiments of the present application (that is, these details should be fully within the scope of understanding of those skilled in the art). In the case of elaborating specific details (e.g., circuits) to describe exemplary embodiments of the present application, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details or when these specific details are changed. Therefore, these descriptions should be considered to be illustrative rather than restrictive.

[0122] Although the present application has been described in conjunction with specific embodiments of the present application, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0123] The embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the present application. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the protection scope of the present application.

Claims

1. An image processing method, characterized in that: include: Collect the image of the environment where the current vehicle is located to obtain the real environment image; Identify the type of environmental elements in the real environment image to obtain at least one type label; Acquire scene images corresponding to each of the type labels from an external database to obtain multiple scene element images; A three-dimensional panoramic image simulating the current vehicle surrounding environment is constructed based on multiple scene element images.

2. An image processing method according to claim 1, characterized in that: The three-dimensional panoramic image simulating the surrounding environment of the current vehicle is constructed based on multiple scene element images, including: Using image stitching technology to stitch the multiple scene element images into a planar panoramic image; The planar panoramic image is mapped into a spherical panoramic image by using a three-dimensional engine technology to obtain the three-dimensional panoramic image.

3. An image processing method according to claim 2, characterized in that: Before mapping the planar panoramic image into a spherical panoramic image using the three-dimensional engine technology, the method further includes: The image object removal technology is used to remove objects in the planar panoramic image that are below a preset height threshold from the ground.

4. The image processing method according to claim 1, characterized in that: Also includes: Obtain the three-dimensional model data of the current vehicle; The three-dimensional model data of the current vehicle is fused with the three-dimensional panoramic image to obtain a three-dimensional panoramic image including the three-dimensional model of the current vehicle.

5. The image processing method according to claim 1, characterized in that: The real environment image includes a vehicle forward environment image, and the method further includes: In response to the type label corresponding to the vehicle forward environment image being in compliance with a preset label range that affects driving safety, a prompt message is generated and issued.

6. An image processing method according to claim 5, characterized in that: The generating and issuing of prompt information comprises: Corresponding warning information and / or driving advice is generated according to the type label, and the warning information and / or driving advice is fused with the three-dimensional panoramic image to obtain a three-dimensional panoramic image containing forward environment prompt information.

7. An image processing method according to claim 1, characterized in that: Also includes: Get the navigation information of the current vehicle; Generate driving direction guidance information based on the navigation information and the real environment image; The driving direction guidance information is fused with the three-dimensional panoramic image to obtain a three-dimensional panoramic image containing the driving direction guidance.

8. An image processing device, characterized in that: include: The acquisition module is used to acquire the image of the environment in which the vehicle is currently located to obtain the real environment image; An element recognition module, used to recognize the type of environmental elements in the real environment image and obtain at least one type label; An image acquisition module, used to acquire scene images corresponding to each type label from an external database to obtain multiple scene element images; The generation module is used to construct a three-dimensional panoramic image simulating the current vehicle surrounding environment based on multiple scene element images.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the program, the image processing method according to any one of claims 1 to 7 is implemented.

10. A vehicle, characterized in that: The vehicle comprises the electronic device as claimed in claim 9.

Images