A vehicle panoramic image generation method, device, equipment, system and medium

Abstract

The embodiment of the specification discloses a vehicle panoramic image generation method, device, equipment, system and medium, a reference distance from a vehicle sensor arranged at a vehicle to a flat ground is measured; a target distance from a target medium is measured by using the vehicle sensor during driving of the vehicle; whether the vehicle is trapped in the target medium is judged, if the vehicle is trapped in the target medium, a target medium plane where a virtual target medium having a mapping relationship with the target medium where the vehicle is trapped is located in a virtual space constructed by a panoramic image system of the vehicle is determined according to the target distance measured by each vehicle sensor; and a part below the target medium plane in the virtual space constructed by the panoramic image system is shielded to avoid the panoramic image presented to the user from being inconsistent with the actual situation, which is beneficial to improving the user experience.

Description

Technical Field

[0001] This application relates to the field of computer data processing technology, and in particular to a method, apparatus and equipment for generating panoramic vehicle images. Background Technology

[0002] In daily life, vehicles not only serve as commuting tools but also meet the needs of self-driving tours and wilderness adventures, greatly facilitating people's daily lives. However, road conditions in self-driving tours or wilderness adventures are often more complex than in urban areas, encountering terrain such as rivers, mud, swamps, and wetlands. Even in urban areas, heavy rainfall can lead to low-lying areas and flooding. When a vehicle drives into deep floodwaters, it may stall, and in more serious cases, endanger the lives of passengers. Therefore, it is crucial for vehicle occupants to be aware of the water conditions outside the vehicle when driving in these dangerous environments.

[0003] Existing technologies only address vehicle wading situations and do not cover methods for handling vehicles in muddy, swampy, or wetland terrain. Currently, sensors or cameras are typically used to detect water depth, and the results are then presented to users visually. For example, patents CN109677266A and CN110843786A depict both the vehicle and the water in three-dimensional or three-dimensional diagrams, which are then displayed to users for reference. Another example is patent CN110254353A, which synthesizes water accumulation information into a transparent chassis (a function of panoramic imaging) and overlays a virtual vehicle to allow users to monitor water accumulation information in real time. However, in existing technologies, both stereoscopic and three-dimensional illustrations are simulated images. Although stereoscopic and three-dimensional illustrations are drawn based on real data, they can never be exactly the same as the actual scene, making the image presented to the user less intuitive. Furthermore, using the transparent chassis function in panoramic imaging to represent wading does not make wading a standard function of panoramic imaging. Moreover, the transparent chassis is superimposed on the virtual vehicle, making the image presented to the user inconsistent with reality, resulting in an unrealistic intuitive experience for the user. Summary of the Invention

[0004] This specification provides a method, apparatus, and device for generating panoramic vehicle images to address the problem of unrealistic panoramic images generated by existing methods.

[0005] To solve the above-mentioned technical problems, the embodiments in this specification are implemented as follows:

[0006] This specification provides an embodiment of a method for generating a panoramic vehicle image, the method including:

[0007] The reference distance is obtained by the vehicle sensor installed at the vehicle location; the reference distance is the distance between the vehicle sensor and the flat ground.

[0008] Acquire the target distance measured by the vehicle's sensors during the vehicle's operation;

[0009] If the target distance measured by the vehicle sensor is less than the reference distance, then based on the target distance measured by the vehicle sensor, a target medium plane in the virtual space constructed by the vehicle's panoramic imaging system is determined to be located in the virtual space where the virtual target medium that the vehicle is trapped is mapped; wherein, the virtual space also includes a virtual vehicle that is mapped to the vehicle based on the reference distance;

[0010] The portion of the virtual vehicle below the target medium plane is occluded to obtain the panoramic image to be displayed to the user.

[0011] This specification provides an embodiment of a vehicle panoramic image generation device, comprising:

[0012] A reference distance acquisition module is used to acquire a reference distance measured by a vehicle sensor installed at the vehicle location; the reference distance is the distance between the vehicle sensor and the flat ground.

[0013] A target distance acquisition module is used to acquire the target distance measured by the vehicle sensors during the vehicle's movement.

[0014] The target medium plane determination module is used to determine, based on the target distance measured by the vehicle sensor, the target medium plane in the virtual space constructed by the vehicle's panoramic imaging system, where a virtual target medium with a mapping relationship to the target medium into which the vehicle is trapped is located, if the target distance measured by the vehicle sensor is less than the reference distance; wherein, the virtual space also includes a virtual vehicle with a mapping relationship to the vehicle determined based on the reference distance;

[0015] The processing module is used to occlude the portion of the virtual vehicle below the target medium plane to obtain the panoramic image to be displayed to the user.

[0016] This specification provides an embodiment of a computer device / equipment / system that may include a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of a method for generating a panoramic vehicle image.

[0017] This specification provides an embodiment of a computer-readable storage medium storing a computer program or instructions that can be executed by a processor to implement a method for generating a panoramic image of a vehicle.

[0018] The embodiments of this specification provide a computer program product that may include a computer program or instructions, which, when executed by a processor, implement the steps of a method for generating a panoramic vehicle image.

[0019] At least one embodiment in this specification can achieve the following beneficial effects: by acquiring a reference distance measured by a vehicle sensor installed at the vehicle; and acquiring a target distance measured by the vehicle sensor during the vehicle's movement; determining whether the vehicle is in a target medium; when the target distance measured by the vehicle sensor is less than the reference distance, it indicates that the vehicle is in the target medium; thereby, based on the target distance measured by the vehicle sensor, determining the target medium plane in the virtual space constructed by the panoramic imaging system of the vehicle, where the virtual target medium with a mapping relationship between the vehicle and the target medium into which the vehicle is trapped is located; and occluding the portion of the virtual vehicle in the panoramic image below the target medium plane to effectively prevent the virtual vehicle in the panoramic image displayed to the user from appearing to float on water or in mud, making the image presented to the user inconsistent with reality, and thus causing the user's intuitive feeling when viewing the panoramic image to be unrealistic. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating a method for generating a panoramic vehicle image provided in the embodiments of this specification;

[0022] Figure 2 This is a schematic diagram of the structure of a vehicle panoramic image generation device provided in the embodiments of this specification;

[0023] Figure 3 This is a schematic diagram of the structure of a vehicle panoramic image generation device provided in the embodiments of this specification. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of one or more embodiments of this specification clearer, the technical solutions of one or more embodiments of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of one or more embodiments of this specification.

[0025] The technical solutions provided in the various embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0026] In existing technologies, when dealing with vehicles wading through water, sensors or cameras are typically used to detect the water depth, and the detection results are then provided to the user in a visual form. For example, patents CN109677266A and CN110843786A draw three-dimensional or three-dimensional diagrams of both the vehicle and the water, and then display the resulting three-dimensional or three-dimensional diagrams to the user for reference. Another example is patent CN110254353A, which synthesizes water accumulation information into a transparent chassis (a function of panoramic imaging) and overlays a virtual vehicle to allow the user to monitor water accumulation information in real time. However, existing technologies, whether 3D or stereoscopic, are all simulated images. Although they are drawn based on real data, they can never be exactly the same as the actual scene, especially when it comes to dynamic elements like water. This makes the images presented to users less intuitive. Furthermore, using the transparent chassis function in panoramic imaging to represent wading does not make wading a standard function of panoramic imaging. Overlaying the transparent chassis with the virtual vehicle makes the tires of the virtual vehicle in the panoramic image appear to be floating on the water in wading scenarios. This results in a discrepancy between the image presented to the user and reality, leading to an unrealistic viewing experience.

[0027] To address the shortcomings of existing technologies, this solution provides the following embodiments:

[0028] Figure 1 This is a flowchart illustrating a method for generating a panoramic vehicle image, as provided in an embodiment of this specification. From a programming perspective, the entity executing the process can be a vehicle server or a processor mounted on the vehicle.

[0029] like Figure 1 As shown, the process may include the following steps:

[0030] Step 102: Obtain the reference distance measured by the vehicle sensor installed at the vehicle location; the reference distance is the distance between the vehicle sensor and the flat ground.

[0031] In the embodiments of this specification, the flat ground can be a flat, level surface without any target medium (e.g., standing water, mud, or sand); the vehicle sensor in the embodiments of this specification can be a sensor capable of measuring distance, such as a ranging sensor, ultrasonic radar, millimeter-wave radar, lidar, or image acquisition equipment, etc. The type and model of the vehicle sensor are not specifically limited here.

[0032] In the embodiments of this specification, multiple vehicle sensors can be installed around the vehicle body, perpendicular to a flat surface, to measure vertical distances from the flat surface. The installation locations of the vehicle sensors can include: below the left rearview mirror, below the right rearview mirror, below the rear bumper, or near the rear license plate, positions where the distance to the flat surface can be measured. Of course, the installation locations of the vehicle sensors can also be other positions where vertical distances to the flat surface can be measured; no specific limitation is made to the installation locations of the vehicle sensors here. Since three points define a plane, the number of the multiple vehicle sensors should be no less than three.

[0033] Step 104: Obtain the target distance measured by the vehicle sensors during the vehicle's operation.

[0034] In the embodiments of this specification, the "driving process," that is, the process of vehicle operation, includes a state where the vehicle is temporarily stopped; the "driving" does not necessarily mean that the vehicle is moving. The target distance can be the distance between each vehicle sensor and the target medium or the ground, measured by each vehicle sensor during the vehicle's operation. The target medium can include a soft target medium, such as water (the water surface can be road surface water, river water, stream water, etc.), swamp, mud, grassland, sand, etc.; or a hard target medium, such as stone. It is understood that the vehicle cannot get stuck in a hard target medium. The type of target medium is not specifically limited here.

[0035] Step 106: If the target distance measured by the vehicle sensor is less than the reference distance, then based on the target distance measured by the vehicle sensor, determine the target medium plane in the virtual space constructed by the panoramic imaging system of the vehicle where the virtual target medium that the vehicle is trapped is located; wherein, the virtual space also includes virtual vehicles that are mapped to the vehicle based on the reference distance.

[0036] In the embodiments of this specification, it can be understood that if the target medium is a soft target medium, the vehicle may get stuck in it, so that the distance measured by the vehicle sensor is not the distance to the real ground but the distance to the target medium, such as the distance to the water surface; thus, the target distance from the target medium measured by the vehicle sensor is less than the reference distance from the real ground.

[0037] In the embodiments of this specification, if the target distance measured by the same vehicle sensor is less than the reference distance, it indicates that the vehicle is trapped in the target medium. The virtual target medium is a portion of the image data displayed in the panoramic image and subsequently shown to the user.

[0038] In the embodiments of this specification, the virtual space can be a 3D space model that already exists in the panoramic imaging system. Before determining the target medium plane in the virtual space constructed by the panoramic imaging system of the vehicle, which has a mapping relationship with the target medium into which the vehicle is trapped, the plane where the lowest point of each tire of the virtual vehicle is located in the virtual space can be calibrated with the flat plane at the reference distance from the vehicle sensor, so that the plane in the virtual space where the flat plane at the reference distance from the vehicle sensor is located and the plane where the lowest point of each tire of the virtual vehicle is located are both the bottom surface of the 3D space model.

[0039] In the embodiments described in this specification, the above calibration process can be completed when generating the panoramic image of the vehicle, or it can be completed before generating the panoramic image; no specific limitation is made here.

[0040] In the embodiments of this specification, the target medium plane where the virtual target medium is located can be used to reflect the plane displayed by the target medium in the virtual space constructed by the vehicle's panoramic imaging system. In the embodiments of this specification, the coordinates of the measurement points of three or more vehicle sensors at the target medium can be used to determine the coordinates of the measurement points of the vehicle sensors at the target medium in the virtual space constructed by the panoramic imaging system, thereby determining the target medium plane where the target medium is located in the virtual space based on the coordinates in the virtual space.

[0041] Step 108: Obscure the portion of the virtual vehicle below the target medium plane to obtain the panoramic image to be displayed to the user.

[0042] In the embodiments of this specification, the portion of the virtual vehicle below the target medium plane is occluded. Specifically, this can be done by not displaying the portion of the virtual vehicle below the target medium plane in the panoramic imaging system, or by displaying it in a semi-transparent or other transparency manner. Of course, different occlusion display options can also be set in the panoramic imaging system for users to choose from.

[0043] In the embodiments of this specification, the portion of the virtual vehicle displayed in the panoramic imaging system that is below the target medium plane (e.g., the wheels and part of the body of the virtual vehicle below the target medium plane) is occluded. This can effectively prevent the virtual vehicle displayed in the panoramic imaging system from appearing to float on the water or in the mud, thus making the image presented to the user inconsistent with reality and resulting in an unrealistic intuitive experience for the user when viewing the image.

[0044] It should be understood that the order of some steps in the methods described in one or more embodiments of this specification may be interchanged according to actual needs, or some steps may be omitted or deleted.

[0045] Figure 1 The method involves acquiring a reference distance measured by vehicle sensors installed at the vehicle location; acquiring a target distance measured by the same vehicle sensors during vehicle operation; determining whether the vehicle is in a target medium; and determining that the vehicle is in a target medium if the target distance measured by the vehicle sensors is less than the reference distance. Based on the target distance measured by the vehicle sensors, the method determines the target medium plane in the virtual space constructed by the panoramic imaging system of the vehicle, which is mapped to the target medium in which the vehicle is located. The method also involves occluding the portion of the virtual vehicle in the panoramic imaging system that is below the target medium plane to effectively prevent the virtual vehicle in the panoramic image displayed to the user from appearing to float on water or in mud, thus avoiding a discrepancy between the presented image and reality and resulting in an unrealistic viewing experience for the user.

[0046] based on Figure 1 In addition to the method described herein, this specification also provides some specific implementation methods of this method, which will be described below.

[0047] In the embodiments of this specification, in order to facilitate understanding of the scheme, a specific process is also provided for determining the target medium plane in the virtual space constructed by the panoramic imaging system of the vehicle, which is a virtual target medium that has a mapping relationship with the target medium into which the vehicle is trapped.

[0048] Optionally, the coordinate transformation relationship between the first coordinate system constructed based on the vehicle and the second coordinate system established based on the virtual vehicle is determined.

[0049] The determination of the virtual target medium, which has a mapping relationship with the target medium into which the vehicle is trapped, and its location in the target medium plane within the virtual space constructed by the vehicle's panoramic imaging system, may specifically include:

[0050] Determine the first position coordinates of multiple first target points at the target medium in the first coordinate system.

[0051] Based on the coordinate transformation relationship and the first position coordinates of the plurality of first target points in the first coordinate system, the second position coordinates of each of the first target points in the second coordinate system are determined.

[0052] Based on the second position coordinates of multiple first target points in the second coordinate system, the target medium plane in the virtual space constructed by the panoramic imaging system of the vehicle is determined to be the virtual target medium that has a mapping relationship with the target medium into which the vehicle is trapped.

[0053] In the embodiments of this specification, the plurality of first target points at the target medium can be a plurality of first target points on the surface of the target medium. Specifically, the plurality of first target points at the target medium can be the projection points of each of the vehicle sensors on the surface of the target medium. Of course, the first target points can also be points on the surface of other target media whose coordinates can be determined, and no specific limitation is made here.

[0054] In the embodiments described in this specification, the vehicle is a vehicle in a real environment. The first coordinate system established based on the vehicle can be any one of the geodetic coordinate system or the world coordinate system. In practical applications, a 3D spatial model (i.e., virtual space) already exists in the panoramic imaging system. The second coordinate system established based on the virtual vehicle can be the coordinate system of the virtual space (3D spatial model) constructed by the panoramic imaging system. The plane where the lowest point of each tire of the virtual vehicle is located in the panoramic imaging system can be a plane determined by any two coordinate axes in the second coordinate system. For example, the plane where the lowest point of each tire of the virtual vehicle is located in the panoramic image can be a plane determined by the X-axis and Y-axis in the second coordinate system.

[0055] In the embodiments of this specification, the coordinate transformation relationship between the first coordinate system and the second coordinate system can be predetermined before acquiring the reference distance measured by each vehicle sensor installed at the vehicle, so as to display the target medium determined based on the target distance data measured by the vehicle sensors in the panoramic imaging system; of course, the coordinate transformation relationship between the first coordinate system and the second coordinate system can also be determined as needed during the generation of the vehicle panoramic image, which is not specifically limited here.

[0056] In the embodiments of this specification, each of the first target points can be measurement points of the vehicle sensors at the target medium at the target distance, and the first position coordinates can be the coordinates of the measurement points of the vehicle sensors at the target medium at the target distance in the first coordinate system. In practical applications, the first position coordinates in the first coordinate system and the second position coordinates in the second coordinate system can be determined according to the coordinate transformation relationship between the first coordinate system and the second coordinate system. Thus, the plane in which the target medium is located in the virtual space constructed by the panoramic imaging system is determined according to the second position coordinates. That is, the target medium plane in the virtual space constructed by the panoramic imaging system of the vehicle is the virtual target medium that has a mapping relationship with the target medium.

[0057] In the embodiments of this specification, in order to facilitate understanding of the scheme, a specific process is also provided for determining the coordinate transformation relationship between the first coordinate system constructed based on the vehicle and the second coordinate system established based on the virtual vehicle.

[0058] Specifically, determining the coordinate transformation relationship between the first coordinate system constructed based on the vehicle and the second coordinate system established based on the virtual vehicle may include:

[0059] The third position coordinates in the first coordinate system are determined by projecting multiple second target points obtained by each marker point set on the vehicle onto the flat ground.

[0060] Determine the fourth position coordinates of the lowest point of each tire of the virtual vehicle in the virtual space in the second coordinate system.

[0061] Based on the third position coordinate in the first coordinate system and the fourth position coordinate in the second coordinate system, determine the coordinate transformation relationship between the first coordinate system and the second coordinate system.

[0062] In the embodiments of this specification, the coordinate transformation relationship can reflect that the plane where each of the second target points is located is the same plane as the plane where the lowest point of each of the tires is located.

[0063] In the embodiments of this specification, the marker points on the vehicle can be the location points of various vehicle sensors, and each of the second target points can be the measurement points obtained by projecting each of the vehicle sensors onto a flat surface at a reference distance; the third position coordinates can be the coordinates of the measurement points of each of the vehicle sensors on the flat surface at the reference distance in a first coordinate system. Of course, the marker points on the vehicle can also be other location points on the vehicle, and the coordinates of the marker points and the measurement points obtained by projecting the marker points onto a flat surface can be obtained using a positioning device.

[0064] In the embodiments of this specification, the fourth position coordinates can be used to reflect the coordinates of the lowest point of each tire of the virtual vehicle in the second coordinate system.

[0065] In the embodiments of this specification, the positions of virtual marker points that are the same as the positions of each of the marker points can be determined on the virtual vehicle in the virtual space. For example, the positions of virtual sensors that are the same as the positions of each of the vehicle sensors can be determined on the virtual vehicle. Based on the determined positions of the virtual sensors, the fifth position coordinates obtained by projecting the virtual sensors onto the plane where the lowest point of the tire is located are determined. Then, the third position coordinates in the first coordinate system are transformed to the fifth position coordinates in the second coordinate system, thereby obtaining the coordinate transformation relationship between the third position coordinates and the fifth position coordinates. The coordinate transformation relationship between the third position coordinates and the fifth position coordinates is then determined as the coordinate transformation relationship between the first coordinate system and the second coordinate system.

[0066] In the embodiments of this specification, the coordinates of the lowest point of each tire of the vehicle at the sixth position in the first coordinate system can also be determined, and then the sixth position coordinates in the first coordinate system can be transformed to the fourth position coordinates in the second coordinate system to obtain the coordinate transformation relationship between the sixth position coordinates and the fourth position coordinates. The coordinate transformation relationship between the sixth position coordinates and the fourth position coordinates can then be determined as the coordinate transformation relationship between the first coordinate system and the second coordinate system.

[0067] In the embodiments of this specification, after determining the coordinate transformation relationship between the first coordinate system and the second coordinate system, the position coordinates of the measurement points of each vehicle sensor at the target medium during vehicle operation can be conveniently and quickly determined in the second coordinate system based on the coordinate transformation relationship between the first coordinate system and the second coordinate system.

[0068] In the embodiments described in this specification, vehicles typically have a maximum safe height that allows the vehicle to sink into water or mud. If the sinking depth exceeds the maximum safe height, the vehicle may stall, or in severe cases, it may pose a danger to the users inside the vehicle.

[0069] Based on this, after determining the target medium plane in the virtual space constructed by the vehicle's panoramic imaging system, which is mapped to the target medium into which the vehicle is trapped, according to the target distance measured by the vehicle sensors, the step may further include:

[0070] Determine the safe height at which the vehicle is allowed to sink into the target medium.

[0071] Based on the coordinate transformation relationship and the safety height, a limit marker that maps to the safety height is marked at the virtual vehicle.

[0072] In the embodiments of this specification, the safe height at which the vehicle can sink into the target medium can usually be preset at the factory. In actual application, if the depth of the vehicle sinking into the target medium exceeds the safe height, it may cause the engine to stall or other adverse effects that damage or affect the vehicle's performance. The safe height at which the vehicle can sink into the target medium may vary for different types of target media.

[0073] In the embodiments of this specification, the type of the target medium can be identified based on the environmental data around the vehicle acquired by the image acquisition device mounted on the vehicle during vehicle operation; specifically, machine learning can be used to identify the type of the target medium into which the vehicle is stuck. Of course, other methods can also be used to identify the type of the target medium into which the vehicle is stuck. The specific identification method and process for the target medium are not specifically limited here.

[0074] In the embodiments of this specification, the safe height at which the vehicle is allowed to sink into the target medium can be the vertical height from the lowest point of the vehicle's wheels. In practical applications, the vertical height from the lowest point of the vehicle's wheels can be determined based on the coordinate transformation relationship between the first and second coordinate systems to the height of the virtual vehicle in the panoramic imaging system. The height of the virtual vehicle in the panoramic image determined above can be used as the limit height at which the virtual vehicle is submerged by the virtual target medium, and a limit mark is displayed at the limit height of the virtual vehicle to remind the user that the vehicle is allowed to be submerged by the target medium.

[0075] In the embodiments of this specification, when the vehicle is in a target medium such as water or mud, the limit height that the vehicle is allowed to sink into the target medium can be marked on the virtual vehicle. This allows the user to easily determine the degree to which the vehicle is submerged in the target medium based on the limit mark displayed on the virtual vehicle and the current target medium plane, and thus determine the distance between the target medium plane and the safety limit mark.

[0076] In the embodiments described in this specification, when the vehicle becomes deeply embedded in the target medium, the user should be promptly alerted to prevent danger from occurring.

[0077] Based on this, after determining the safe height at which the vehicle is allowed to sink into the target medium, the process may further include:

[0078] Based on the target distance measured by the vehicle sensors, the permissible height from the target medium to the safe height is determined.

[0079] Determine the target threshold range within which the allowed height falls.

[0080] Based on the target threshold range, generate alarm information with an alarm level corresponding to the target threshold range.

[0081] The alarm information is displayed in the panoramic imaging system.

[0082] In the embodiments of this specification, the height at which the vehicle is trapped in the target medium can be determined by subtracting the reference distance measured by each of the vehicle sensors from the target distance measured by each of the vehicle sensors. Then, the allowable height can be determined based on the safe height and the height at which the vehicle is trapped in the target medium.

[0083] In the embodiments of this specification, the allowable height from the target medium to the safe height can be divided into different threshold ranges. Different hazard levels and alarm levels can be set for different threshold ranges. The smaller the endpoint value of the threshold range, the deeper the vehicle is trapped in the target medium, and the higher the hazard level. That is, the smaller the allowable height, the higher the hazard level. In this application, the smaller the endpoint value of the target threshold range, the higher the alarm level corresponding to the target threshold range. For example, the hazard level can be set to level three, so different alarm forms (i.e., alarm levels) can be set for different hazard levels. By generating alarm information to prompt the user, the user can be effectively reminded to prevent danger from occurring.

[0084] In practical applications, alarm information can be generated using sound, light, electricity, or other methods. Specifically, the alarm information may include one or more of the following: alarm sound, voice reminder, and flashing limit indicator. No specific limitations are made here regarding the specific alarm type and alarm content.

[0085] In the embodiments of this specification, in order to facilitate users to know the depth of the vehicle stuck in the target medium in a timely manner, the panoramic imaging system can be set to start automatically when the depth of the vehicle stuck in the target medium reaches a preset value.

[0086] Based on this, after obtaining the target distance measured by the vehicle sensors during the vehicle's operation, the process may further include:

[0087] The depth to which the vehicle is embedded in the target medium is determined based on the target distance and the reference distance.

[0088] Determine whether the depth is greater than the preset height, and obtain the determination result.

[0089] If the determination result indicates that the depth is greater than the preset height, then the panoramic imaging system is activated.

[0090] In the embodiments of this specification, when the vehicle is stuck in the target medium at a depth greater than a preset height, an instruction can be generated to control the panoramic imaging system to open or turn on, so that the panoramic imaging system can be automatically turned on and an alarm message can be generated to alert the user that the vehicle is in danger, so that the user can know the specific situation of the vehicle being stuck in the target medium at the first time and avoid the trouble of manually turning on the panoramic imaging system.

[0091] In the embodiments of this specification, due to the occlusion processing performed on the portion of the virtual vehicle in the panoramic imaging system that is below the target medium plane, the virtual vehicle near the target medium plane may appear too rigid and unrealistic.

[0092] Based on this, after the portion of the virtual vehicle below the target medium plane is occluded, it may further include:

[0093] The portion of the virtual vehicle located at the target medium plane and / or within a preset range above the target medium plane is optimized to obtain the panoramic image to be displayed to the user.

[0094] The optimization process includes any one or more of the following: fading out, blurring, or overlaying water splash animation.

[0095] In the embodiments of this specification, the target medium plane can be used to reflect the boundary between the hidden and unhidden parts of the virtual vehicle; the target medium plane can be a water surface; in practical applications, visual optimization processing can be performed on the boundary between the hidden and unhidden parts of the virtual vehicle (e.g., at the target medium plane, and a preset range above the target medium plane), so that the boundary between the virtual vehicle and the target medium plane is not a hard transition, giving the user a more realistic visual experience; the optimization processing can be fading and / or blurring processing of the virtual vehicle at the boundary.

[0096] In practical applications, when the target medium is a water surface, the splashing water animation can be superimposed on the boundary between the hidden and unhidden parts of the virtual vehicle based on the vehicle speed and direction of travel, thus making the visual experience more realistic for the user.

[0097] In practical applications, the optimization process may also include other processes that make the user's visual experience more realistic, which are not specifically limited here.

[0098] In the embodiments described in this specification, in order to display more realistic image data in the panoramic imaging system, the environmental data of the vehicle's location can be displayed in the panoramic imaging system.

[0099] Based on this, before performing the occlusion process on the portion of the virtual vehicle below the target medium plane, the process may further include:

[0100] The environmental data around the vehicle is acquired from the various cameras located at the vehicle.

[0101] The environmental data around the vehicle captured by each of the camera devices are stitched together to obtain environmental image data.

[0102] The environmental image data and the virtual vehicle model are overlaid and displayed in the panoramic imaging system.

[0103] In the embodiments of this specification, the imaging device may be a camera, and the camera may be an ultra-wide-angle fisheye lens; in the embodiments of this specification, multiple ultra-wide-angle fisheye lenses can be used to capture environmental data around the vehicle, and then distortion correction and stitching are performed on the captured environmental data around the vehicle to form environmental image data around the vehicle.

[0104] In the embodiments of this specification, the environmental image data of the vehicle's surroundings formed above can be displayed in the panoramic imaging system and superimposed on the virtual vehicle in the panoramic imaging system, thereby making the user's visual experience of viewing the panoramic image more realistic and making it easier to judge the surrounding environment of the vehicle.

[0105] The above method involves obtaining a reference distance measured by vehicle sensors installed at the vehicle location, and a target distance measured by the same vehicle sensors during vehicle operation. This allows for the determination of whether the vehicle is in a target medium. If the target distance measured by the vehicle sensors is less than the reference distance, it indicates that the vehicle is in the target medium. Based on the target distances measured by each vehicle sensor, the target medium plane of the virtual target medium, which has a mapping relationship with the target medium the vehicle is in, is determined within the virtual space constructed by the vehicle's panoramic imaging system. Furthermore, the portion of the virtual vehicle below the target medium plane is occluded to effectively prevent the virtual vehicle in the panoramic image displayed to the user from appearing to float on water or in mud, thus avoiding a discrepancy between the displayed image and reality and resulting in an unrealistic viewing experience for the user.

[0106] Based on the same idea, embodiments of this specification also provide apparatus corresponding to the above methods. Figure 2 The embodiments provided in this specification correspond to Figure 1 A schematic diagram of the structure of a vehicle panoramic imaging device. (See diagram below.) Figure 2 As shown, the device may include:

[0107] The reference distance acquisition module 202 is used to acquire the reference distance measured by the vehicle sensor installed at the vehicle; the reference distance is the distance between the vehicle sensor and the flat ground.

[0108] The target distance acquisition module 204 is used to acquire the target distance measured by the vehicle sensors during the vehicle's operation.

[0109] The target medium plane determination module 206 is used to determine, based on the target distance measured by the vehicle sensor, the target medium plane in the virtual space constructed by the panoramic imaging system of the vehicle, where a virtual target medium with a mapping relationship to the target medium into which the vehicle is trapped is located, if the target distance measured by the vehicle sensor is less than the reference distance; wherein, the virtual space also includes a virtual vehicle with a mapping relationship to the vehicle determined based on the reference distance.

[0110] The processing module 208 is used to perform occlusion processing on the part of the virtual vehicle below the target medium plane to obtain the panoramic image to be displayed to the user.

[0111] based on Figure 2 The embodiments of this specification also provide some specific implementation schemes of the method, which are described below.

[0112] Optional, Figure 2 The device may further include:

[0113] The coordinate transformation relationship determination module is used to determine the coordinate transformation relationship between the first coordinate system constructed based on the vehicle and the second coordinate system established based on the virtual vehicle.

[0114] The target medium plane determination module 206 can be specifically used for:

[0115] Determine the first position coordinates of multiple first target points at the target medium in the first coordinate system.

[0116] Based on the coordinate transformation relationship and the first position coordinates of the plurality of first target points in the first coordinate system, the second position coordinates of each of the first target points in the second coordinate system are determined.

[0117] Based on the second position coordinates of multiple first target points in the second coordinate system, the target medium plane in the virtual space constructed by the panoramic imaging system of the vehicle is determined to be the virtual target medium that has a mapping relationship with the target medium into which the vehicle is trapped.

[0118] Optionally, the coordinate transformation relationship determination module can be specifically used for:

[0119] The third position coordinates in the first coordinate system are determined by projecting multiple second target points obtained by each marker point set on the vehicle onto the flat ground.

[0120] Determine the fourth position coordinates of the lowest point of each tire of the virtual vehicle in the virtual space in the second coordinate system.

[0121] Based on the third position coordinate in the first coordinate system and the fourth position coordinate in the second coordinate system, determine the coordinate transformation relationship between the first coordinate system and the second coordinate system.

[0122] Optional, Figure 2 The device may further include:

[0123] The safe height determination module is used to determine the safe height at which the vehicle is allowed to sink into the target medium.

[0124] The limit marking module is used to mark a limit mark at the virtual vehicle that has a mapping relationship with the safety height, based on the coordinate transformation relationship and the safety height.

[0125] Optional, Figure 2 The device may further include:

[0126] The allowable height determination module is used to determine the allowable height from the target medium to the safe height based on the target distance measured by the vehicle sensors.

[0127] The target threshold range determination module is used to determine the target threshold range in which the allowable height falls.

[0128] An alarm notification generation module is used to generate alarm information corresponding to the alarm level of the target threshold range based on the target threshold range.

[0129] The display module is used to display the alarm information in the panoramic imaging system.

[0130] Optional, Figure 2 The device may further include:

[0131] The depth determination module is used to determine the depth at which the vehicle is embedded in the target medium based on the target distance and the reference distance.

[0132] The judgment module is used to determine whether the depth is greater than a preset height and obtain a judgment result.

[0133] The startup module is used to start the panoramic imaging system if the judgment result indicates that the depth is greater than the preset height.

[0134] Optional, Figure 2 The device may further include:

[0135] The optimization module is used to optimize the portion of the virtual vehicle located at the target medium plane and / or within a preset range above the target medium plane to obtain the panoramic image to be displayed to the user.

[0136] The optimization process includes any one or more of the following: fading out, blurring, or overlaying water splash animation.

[0137] Optional, Figure 2 The device may further include:

[0138] An environmental data acquisition module is used to acquire environmental data around the vehicle captured by various cameras at the vehicle.

[0139] The stitching module is used to stitch together the environmental data around the vehicle captured by the various cameras to obtain environmental image data.

[0140] The display module is used to overlay the environmental image data and the vehicle virtual model onto the panoramic image system.

[0141] Based on the same idea, this specification also provides devices corresponding to the above methods in its embodiments.

[0142] Figure 3 The embodiments provided in this specification correspond to Figure 1 A schematic diagram of the structure of a vehicle panoramic image generation device. (See diagram below.) Figure 3 As shown, device 300 may include:

[0143] The system includes a memory 330, a processor 310, and a computer program 320 stored in the memory. The processor 310 executes the computer program 320 to implement the steps of the vehicle panoramic image generation method described in any of the above embodiments.

[0144] Based on the same approach, embodiments of this specification also provide a computer-readable storage medium corresponding to the above-described method. The computer-readable storage medium stores a computer program or instructions that can be executed by a processor to implement the above-described method for generating panoramic vehicle images.

[0145] Based on the same approach, embodiments of this specification also provide a computer program product corresponding to the above-described method. The computer program product includes a computer program or instructions that, when executed by a processor, can implement the steps of the above-described vehicle panoramic image generation method.

[0146] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on its differences from other embodiments. In particular, for... Figure 3 As the device shown is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0147] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0148] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0149] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0150] For ease of description, the above devices are described separately by function as various units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0151] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0152] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0153] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0154] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0155] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0156] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0157] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0158] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0159] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0160] This application can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0161] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for generating a panoramic image of a vehicle, characterized in that, The method includes: The reference distance is obtained by measuring the distance between the vehicle sensor and the flat ground; the reference distance is the distance between the vehicle sensor and the flat ground. Acquire the target distance measured by the vehicle's sensors during the vehicle's operation; If the target distance measured by the vehicle sensor is less than the reference distance, then based on the target distance measured by the vehicle sensor, a target medium plane in the virtual space constructed by the vehicle's panoramic imaging system is determined to be located in the virtual space where the virtual target medium that the vehicle is trapped is mapped; wherein, the virtual space also includes a virtual vehicle that is mapped to the vehicle based on the reference distance; The portion of the virtual vehicle below the target medium plane is occluded to obtain the panoramic image to be displayed to the user.

2. The method according to claim 1, characterized in that, The method further includes: Determine the coordinate transformation relationship between the first coordinate system constructed based on the vehicle and the second coordinate system established based on the virtual vehicle; The determination of the virtual target medium, which has a mapping relationship with the target medium into which the vehicle is trapped, is located in the target medium plane of the virtual space constructed by the vehicle's panoramic imaging system, including: Determine the first position coordinates of multiple first target points at the target medium in the first coordinate system; Based on the coordinate transformation relationship and the first position coordinates of the plurality of first target points in the first coordinate system, determine the second position coordinates of each first target point in the second coordinate system; Based on the second position coordinates of multiple first target points in the second coordinate system, the target medium plane in the virtual space constructed by the panoramic imaging system of the vehicle is determined to be the virtual target medium that has a mapping relationship with the target medium into which the vehicle is trapped.

3. The method according to claim 2, characterized in that, Determining the coordinate transformation relationship between the first coordinate system constructed based on the vehicle and the second coordinate system established based on the virtual vehicle includes: Determine the third position coordinates in the first coordinate system of multiple second target points obtained by projecting each marker point set on the vehicle onto the flat ground; Determine the fourth position coordinates of the lowest point of each tire of the virtual vehicle in the virtual space in the second coordinate system; Based on the third position coordinate in the first coordinate system and the fourth position coordinate in the second coordinate system, determine the coordinate transformation relationship between the first coordinate system and the second coordinate system.

4. The method according to claim 2, characterized in that, After determining the target medium plane in the virtual space constructed by the vehicle's panoramic imaging system, based on the target distance measured by the vehicle sensors and the virtual target medium that has a mapping relationship with the target medium into which the vehicle is trapped, the method further includes: Determine the safe height at which the vehicle is allowed to sink into the target medium; Based on the coordinate transformation relationship and the safety height, a limit marker that has a mapping relationship with the safety height is marked at the virtual vehicle.

5. The method according to claim 4, characterized in that, After determining the safe height at which the vehicle is allowed to sink into the target medium, the method further includes: Based on the target distance measured by the vehicle sensors, the permissible height from the target medium to the safe height is determined; Determine the target threshold range within which the allowed height falls; Based on the target threshold range, generate alarm information corresponding to the alarm level of the target threshold range; The alarm information is displayed in the panoramic imaging system.

6. The method according to claim 1, characterized in that, After acquiring the target distance measured by the vehicle sensors during the vehicle's operation, the process further includes: The depth to which the vehicle is embedded in the target medium is determined based on the target distance and the reference distance; Determine whether the depth is greater than a preset height, and obtain the determination result; If the determination result indicates that the depth is greater than the preset height, then the panoramic imaging system is activated.

7. The method according to claim 1, characterized in that, After occluding the portion of the virtual vehicle below the target medium plane, the method further includes: The portion of the virtual vehicle located at the target medium plane and / or within a preset range above the target medium plane is optimized to obtain the panoramic image to be displayed to the user. The optimization process includes any one or more of the following: fading out, blurring, or overlaying water splash animation.

8. The method according to claim 1, characterized in that, Before occluding the portion of the virtual vehicle below the target medium plane, the method further includes: Acquire environmental data surrounding the vehicle captured by each camera device at the vehicle location; The environmental data around the vehicle captured by each of the camera devices are stitched together to obtain environmental image data; The environmental image data and the virtual vehicle are overlaid and displayed in the panoramic imaging system.

9. A vehicle panoramic image generation device, characterized in that, The device includes: A reference distance acquisition module is used to acquire a reference distance measured by a vehicle sensor installed at the vehicle location; the reference distance is the distance between the vehicle sensor and the flat ground. A target distance acquisition module is used to acquire the target distance measured by the vehicle sensors during the vehicle's movement. The target medium plane determination module is used to determine, based on the target distance measured by the vehicle sensor, the target medium plane in the virtual space constructed by the vehicle's panoramic imaging system, where a virtual target medium with a mapping relationship to the target medium into which the vehicle is trapped is located, if the target distance measured by the vehicle sensor is less than the reference distance; wherein, the virtual space also includes a virtual vehicle with a mapping relationship to the vehicle determined based on the reference distance; The processing module is used to occlude the portion of the virtual vehicle below the target medium plane to obtain the panoramic image to be displayed to the user.

10. A computer device / equipment / system, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 7.

11. A computer-readable storage medium having a computer program or instructions stored thereon, the computer program or instructions being executable by a processor to implement the vehicle panoramic image generation method of any one of claims 1 to 7.

12. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 7.

Images