A scene reconstruction method, device, vehicle-mounted equipment and computer program product

Abstract

The embodiments of the present invention disclose a scene reconstruction method, apparatus, vehicle-mounted device, and computer program product, including: obtaining the current position data of the target object, converting the current position data to obtain the converted current data to be rendered, determining the current offset data based on the current data to be rendered and the next data to be rendered, determining the rendering coordinates of the current data to be rendered based on the current data to be rendered, the current offset data, and the current offset coefficient, wherein the current offset coefficient is determined based on the difference between the current time of the renderer and a preset time point, wherein the preset time point is the time point between the moment the current position data is obtained and the current time of the renderer, rendering the rendering coordinates of the current data to be rendered, and displaying the obtained display image to achieve scene reconstruction of the target object. In this way, the smoothness of the display image obtained by the vehicle-mounted device rendering the collected data is improved.

Description

Technical Field

[0001] The present invention relates to a vehicle scene reconstruction technology, and in particular to a scene reconstruction method, device, vehicle-mounted equipment and computer program product. Background Art

[0002] At present, with the continuous development of social economy and technological progress, the cockpits of new energy passenger vehicles are becoming more and more intelligent, and the car screens are no longer limited to simple information display and basic operation control in the traditional sense.

[0003] Among them, environmental reconstruction is an application that uses the vehicle's integrated sensors to perceive the environment around the vehicle in real time, reconstruct important environmental elements in 3D and display them on the car screen. This application can provide a more interactive and immersive driving experience.

[0004] However, achieving smooth and accurate real-time display of environmental elements in relative motion generally requires sensors that provide high-precision and high-frequency positional data. However, some mid-range and low-end in-vehicle platforms lack the corresponding high-specification sensor hardware, resulting in poor display smoothness. Summary of the Invention

[0005] Embodiments of the present invention provide a scene reconstruction method, apparatus, vehicle-mounted device, and computer program product, which can improve the smoothness of a display screen obtained by rendering collected data by the vehicle-mounted device.

[0006] The technical solution of the present invention is achieved as follows:

[0007] An embodiment of the present invention provides a scene reconstruction method, comprising:

[0008] Get the current location data of the target object;

[0009] Converting the current position data to obtain converted current data to be rendered;

[0010] Determining current offset data according to the current data to be rendered and the next data to be rendered;

[0011] Determining rendering coordinates of the current data to be rendered based on the current data to be rendered, the current offset data, and a current offset coefficient; wherein the current offset coefficient is determined based on a difference between a current time of the renderer and a preset time point; the preset time point is a time point between the time when the current position data is acquired and the current time of the renderer;

[0012] Rendering coordinates of the current data to be rendered are rendered to obtain a display image, so as to achieve scene reconstruction of the target object.

[0013] In this way, by acquiring the current position data of the target object collected and converting it, the current data to be rendered is obtained, and then the current offset data determined by the current data to be rendered and the subsequent data to be rendered are used, and the rendering coordinates of the current data to be rendered are determined by using the current offset coefficient, so that the updated rendering coordinates of the current data to be rendered are related to the position data obtained twice before and after, so that the rendering coordinates of the current data to be rendered are closer to the real picture when rendered and displayed, and the current offset coefficient is determined based on the difference between the current time of the renderer and the preset time point, wherein the preset time point is: the time point between the acquisition time of the current position data and the current time of the renderer, so that the current offset coefficient is related to the current time of the renderer and the difference between the time point between the acquisition time of the current position data and the current time of the renderer, so that the rendering coordinates of the current data to be rendered are drawn based on the data collected by the components with the data offset added, thereby making the rendered picture closer to the real picture, further improving the smoothness of the displayed picture.

[0014] Furthermore, the current data to be rendered is a coordinate structure defined by the renderer. Accordingly, determining the current offset data according to the current data to be rendered and the next data to be rendered includes:

[0015] Determine a filter structure corresponding to the current rendering data; the filter structure includes: a coordinate structure and a weight value corresponding to the coordinate structure;

[0016] Filtering the filter structure corresponding to the current data to be rendered to obtain filtered coordinates corresponding to the current data to be rendered;

[0017] Current offset data is determined according to the filtered coordinates corresponding to the current data to be rendered and the filtered coordinates corresponding to the next data to be rendered.

[0018] In this way, the current data to be rendered is stored in a coordinate structure, converted into a filter structure, and filtered, so that the current offset data can be obtained based on the filtered coordinates, providing data for determining the rendering coordinates of the current data to be rendered.

[0019] Furthermore, filtering the filter structure corresponding to the current data to be rendered to obtain filtered coordinates corresponding to the current data to be rendered includes:

[0020] Filtering the filter structure corresponding to the current data to be rendered and the filter structures corresponding to the next N data to be rendered of the current data to be rendered to obtain filtered coordinates corresponding to the current data to be rendered;

[0021] Wherein, N is a positive integer greater than or equal to 1.

[0022] In this way, filtering is performed through the filtering structure corresponding to the current data to be rendered and the filtering structure corresponding to the next N data to be rendered, so that the filtered coordinates corresponding to the current data to be rendered are associated with the filtering structures corresponding to the next N data to be rendered, thereby improving the continuity of the display screen through filtering.

[0023] Furthermore, determining the rendering coordinates of the current data to be rendered according to the current data to be rendered, the current offset data, and the current offset coefficient includes:

[0024] forming an interpolation frame structure corresponding to the current data to be rendered by using the filtered coordinates corresponding to the current data to be rendered, the current offset data, and the current offset coefficient;

[0025] The rendering coordinates of the data to be rendered are drawn according to the interpolation frame structure corresponding to the data to be rendered.

[0026] In this way, by determining the interpolation structure of the current data to be rendered as described above, the renderer can draw the rendering coordinates of the current data to be rendered based on the interpolation structure. The elements in the interpolation structure are used to correct the filtered coordinates corresponding to the current data to be rendered, so that the rendering coordinates of the current data to be rendered can be drawn by the renderer with the data offset increased, and the rendered picture is closer to the real picture.

[0027] Furthermore, drawing the rendering coordinates of the current data to be rendered according to the interpolation frame structure corresponding to the current data to be rendered includes:

[0028] Determine a product of the current offset coefficient and the current offset data;

[0029] The sum of the product and the filtered coordinates corresponding to the current data to be rendered is determined as the rendering coordinates of the current data to be rendered.

[0030] In this way, in an embodiment of the present invention, the rendering coordinates of the current data to be rendered are obtained by correcting the filtered coordinates corresponding to the current data to be rendered, so that the rendering coordinates of the current data to be rendered are drawn based on the data collected by the components with the data offset added, thereby improving the smoothness of the picture while making the picture closer to the real picture.

[0031] Furthermore, the method further comprises:

[0032] The proportional coefficient between the difference and the preset interpolation time length is determined as the current offset coefficient.

[0033] In this way, the current offset coefficient is determined by the ratio of the above-mentioned difference value to the preset interpolation duration, so that the determined current offset coefficient is inversely proportional to the preset interpolation duration, which can make the rendering coordinates of the current data to be rendered take into account the preset interpolation duration, thereby improving the accuracy of the rendering coordinates of the current data to be rendered.

[0034] Furthermore, the method further comprises:

[0035] The preset sampling interval duration for the components of the vehicle is determined as the preset interpolation duration.

[0036] In this way, the current offset coefficient is calculated by the above method, so that the current offset coefficient is inversely proportional to the sampling interval duration preset for the vehicle components, so that the rendering coordinates of the current data to be rendered can be determined taking into account the sampling interval duration preset for the vehicle components, thereby improving the accuracy of the rendering coordinates of the current data to be rendered.

[0037] Furthermore, the method further comprises:

[0038] The interval between the acquisition time of the current position data and the acquisition time of the previous position data is determined as the preset interpolation duration.

[0039] In this way, the current offset coefficient is calculated by the above method, so that the current offset coefficient is inversely proportional to the interval between the acquisition times of the two previous and next position data. This enables the rendering coordinates of the current data to be rendered to be determined while taking into account the interval between the acquisition time of the current position data and the acquisition time of the previous position data, thereby improving the accuracy of the rendering coordinates of the current data to be rendered.

[0040] Furthermore, the method further comprises:

[0041] The time point of the renderer after determining the current offset data is determined as the preset time point.

[0042] In this way, the preset time point is determined by the above method, so that the determined difference can better reflect the time interval between the determined current offset data and the rendering of the renderer, which is conducive to determining the rendering coordinates of the current data to be rendered together with the current offset data, so that the rendering coordinates of the current data to be rendered are rendered. The display picture obtained by rendering is closer to the real picture at the current time.

[0043] An embodiment of the present invention provides a scene reconstruction device, comprising:

[0044] An acquisition module is used to obtain the current location data of the target object;

[0045] A conversion module, configured to convert the current position data to obtain converted current data to be rendered;

[0046] a determination module, configured to determine current offset data based on the current data to be rendered and the previous data to be rendered; wherein the current offset coefficient is determined based on a difference between a current time of the renderer and a preset time point; the preset time point being a time point between a time when the current position data is acquired and the current time of the renderer;

[0047] An updating module, configured to determine a rendering coordinate of the current data to be rendered according to the current data to be rendered, the current offset data, and a current offset coefficient;

[0048] The scene reconstruction module is used to render the rendering coordinates of the current data to be rendered through a renderer to obtain a display image, so as to realize scene reconstruction of the target object.

[0049] An embodiment of the present invention provides a vehicle-mounted device, comprising: a processor and a storage medium storing instructions executable by the processor, wherein the storage medium relies on the processor to perform operations through a communication bus, and when the instructions are executed by the processor, the scene reconstruction method described in one or more of the above embodiments is executed.

[0050] An embodiment of the present invention further provides a computer program product, comprising a computer program or instructions, characterized in that when the computer program or instructions are executed by a processor, the steps of the scene reconstruction method described in one or more of the above embodiments are implemented.

[0051] Beneficial effects of the present invention:

[0052] (1) The current position data of the target object is acquired and converted to obtain the current data to be rendered. The current offset data determined by the current data to be rendered and the next data to be rendered are used together with the current offset coefficient to determine the rendering coordinates of the current data to be rendered. The updated rendering coordinates of the current data to be rendered are correlated with the position data acquired twice before and after. The rendering coordinates of the current data to be rendered are rendered to display a screen closer to the real screen.

[0053] (2) The current offset coefficient is determined based on the difference between the current time of the renderer and the preset time point, wherein the preset time point is: the time point between the acquisition time of the current position data and the current time of the renderer, so that the current offset coefficient is related to the current time of the renderer and the difference between the acquisition time of the current position data and the current time of the renderer, so that the rendering coordinates of the current data to be rendered are based on the data collected by the components and the data offset is added, so that the rendered picture is closer to the real picture, and the smoothness of the display picture is further improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 A schematic flow chart of an optional scene reconstruction method provided in an embodiment of the present invention;

[0055] Figure 2 A hardware schematic diagram of an optional execution scene reconstruction method provided by an embodiment of the present invention Figure 1 ;

[0056] Figure 3 A hardware schematic diagram of an optional execution scene reconstruction method provided by an embodiment of the present invention Figure 2 ;

[0057] Figure 4 A hardware schematic diagram of an optional execution scene reconstruction method provided by an embodiment of the present invention Figure 3 ;

[0058] Figure 5 A hardware schematic diagram of an optional execution scene reconstruction method provided by an embodiment of the present invention Figure 4 ;

[0059] Figure 6 A schematic structural diagram of an optional scene reconstruction device provided in an embodiment of the present invention;

[0060] Figure 7 A schematic structural diagram of an optional vehicle-mounted device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0061] The technical solutions in the embodiments of the present invention will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present invention.

[0062] In view of the problem in the related art that the sampling frequency of components of vehicle-mounted equipment is low, resulting in an unsmooth display during scene reconstruction, an embodiment of the present invention provides a scene reconstruction method. Figure 1 A flow chart of an optional scene reconstruction method provided by an embodiment of the present invention is shown as follows: Figure 1 As shown, the scene reconstruction method may include:

[0063] S101: Acquire the current location data of the target object;

[0064] For vehicles, the current location data of target objects can be collected through components on the vehicle body. For example, components such as cameras and sensors can collect the current location data of surrounding vehicles. These components have a fixed sampling frequency. For example, the current location data of target objects can be collected every 10 seconds. Furthermore, this current location data is the location awareness data of scene reconstruction elements in its original format.

[0065] The target object may be a vehicle or other moving object, such as a person or an animal, and this is not specifically limited in the embodiment of the present invention.

[0066] It should be noted that the above-mentioned target object can be one or more. When there are multiple target objects, the current position data of each target object can be collected, so that the scene reconstruction method provided by the embodiment of the present invention can be executed for the current position data of each target object, so as to obtain the display screen of each target object and display it.

[0067] In addition, the vehicle-mounted device may also record the time when the current position data of the target object is obtained this time, thereby recording the time when the current position data of the target object is obtained each time for subsequent determination of the current offset data.

[0068] S102: Convert the current position data to obtain converted current data to be rendered;

[0069] After obtaining the current position data of the target object through the above S101, the current position data can be converted in S102 so that the converted current data to be rendered is data in the 3D coordinate format used by the renderer, which is conducive to rendering and thus obtaining a display screen.

[0070] In this way, for each target object, the current position data can be converted to obtain the current data to be rendered for each target object.

[0071] S103: Determine current offset data based on current data to be rendered and subsequent data to be rendered;

[0072] After obtaining the current data to be rendered of the target object through S102 above, in S103, the on-board device determines the current offset data based on the current data to be rendered and the subsequent data to be rendered. Here, the current data to be rendered and the subsequent data to be rendered can be subtracted to obtain the current offset data. The subsequent data to be rendered is: the data to be rendered obtained by converting the position data collected after the current data to be rendered, and the current data to be rendered and the subsequent data to be rendered are data at two points in time, and the time point of the current data to be rendered is before the time point of the subsequent data to be rendered. In addition, the current data to be rendered and the subsequent data to be rendered are both obtained by converting the position data collected before and after. Since each data to be rendered is a three-dimensional coordinate, the subtraction of the current data to be rendered and the subsequent data to be rendered here refers to the subtraction of the current three-dimensional coordinate and the subsequent three-dimensional coordinate.

[0073] Here, it should be noted that in the vehicle-mounted device, a first-in-first-out queue of length 2 can be used for each target object. In this way, the current data to be rendered can be stored at the head of the queue, and the next data to be rendered can be stored at the end of the queue. In this way, when the current data to be rendered is obtained at the head of the queue, the current offset data can be determined based on the difference between the current data to be rendered and the next data to be rendered at the end of the queue, so that the current offset data can be determined for each target object.

[0074] S104: Determine the rendering coordinates of the current data to be rendered according to the current data to be rendered, the current offset data, and the current offset coefficient;

[0075] After obtaining the current offset data of each target object in S103, in S104, the rendering coordinates of the current data to be rendered are determined based on the current data to be rendered, the current offset data, and the current offset coefficient. The current offset coefficient is determined based on the difference between the current time of the renderer and a preset time point. That is, after obtaining the current data to be rendered and the current offset data, the current offset coefficient can be determined based on the difference between the current time of the renderer and a preset time point, wherein the preset time point is a time point between the moment the current position data is obtained and the current time of the renderer.

[0076] As can be seen, after obtaining the current data to be rendered and the current offset data, the difference between the current time of the renderer and the time point between the acquisition of the current position data and the current time of the renderer is first determined, and then the current offset coefficient is determined based on this difference. Here, the current offset coefficient can be determined using the corresponding relationship between the difference and the offset coefficient, and the current offset coefficient can also be calculated using a preset calculation formula. Here, the embodiments of the present invention do not specifically limit this.

[0077] S105: Rendering the rendering coordinates of the current data to be rendered, obtaining a display image for display, and realizing scene reconstruction of the target object.

[0078] After obtaining the rendering coordinates of the current data to be rendered through the above S104, in S105, the rendering coordinates of the current data to be rendered can be rendered by the renderer to obtain a display screen, and the display screen is displayed, thereby achieving scene reconstruction of the target object.

[0079] In this way, through the above-mentioned display method, even when the acquisition frequency of components is low, the display image of each frame can still be determined and displayed at the display frame rate, while the smoothness of the display image of the vehicle-mounted equipment is improved without the need to replace components.

[0080] Regarding the current offset coefficient, in an optional embodiment, the method may further include:

[0081] The proportional coefficient between the difference and the preset interpolation time length is determined as the current offset coefficient.

[0082] It can be understood that in order to calculate the current offset coefficient, in the embodiment of the present invention, the proportional coefficient between the difference value and the preset interpolation duration can be used as the current offset coefficient.

[0083] The preset interpolation time length may be a preset value or a value calculated in real time, which is not specifically limited in the embodiment of the present invention.

[0084] In this way, the current offset coefficient is determined by the ratio of the above-mentioned difference value to the preset interpolation duration, so that the determined current offset coefficient is inversely proportional to the preset interpolation duration, which can make the rendering coordinates of the current data to be rendered take into account the preset interpolation duration, thereby improving the accuracy of the rendering coordinates of the current data to be rendered.

[0085] Regarding the preset interpolation duration, in an optional embodiment, the method may further include:

[0086] The preset sampling interval duration for the components of the vehicle is determined as the preset interpolation duration.

[0087] Understandably, each vehicle component has a preset sampling interval duration, which is equal to the inverse of the sampling frequency. Once the preset sampling interval duration is determined, the ratio of the difference to the sampling interval duration can be used to determine the current offset coefficient, which is used to determine the rendering coordinates of the current data to be rendered.

[0088] In this way, the current offset coefficient is calculated by the above method, so that the current offset coefficient is inversely proportional to the sampling interval duration preset for the vehicle components, so that the rendering coordinates of the current data to be rendered can be determined taking into account the sampling interval duration preset for the vehicle components, thereby improving the accuracy of the rendering coordinates of the current data to be rendered.

[0089] Regarding the preset interpolation duration, in an optional embodiment, the method may further include:

[0090] The interval between the acquisition time of the current position data and the acquisition time of the previous position data is determined as the preset interpolation duration.

[0091] It is understandable that there is a preset sampling interval duration corresponding to the components of the vehicle, but the sampling interval duration cannot reflect the actual interval duration between the two previous and subsequent data. Here, after the current position data is obtained, the acquisition time of the current position data is recorded. In this way, the acquisition time of the two previous and subsequent position data can be known, and the interval time between the acquisition time of the current position data and the acquisition time of the previous position data is used as the preset interpolation duration.

[0092] In this way, the current offset coefficient is calculated by the above method, so that the current offset coefficient is inversely proportional to the interval between the acquisition times of the two previous and next position data. This enables the rendering coordinates of the current data to be rendered to be determined while taking into account the interval between the acquisition time of the current position data and the acquisition time of the previous position data, thereby improving the accuracy of the rendering coordinates of the current data to be rendered.

[0093] Regarding the above-mentioned preset time point, in an optional embodiment, the above-mentioned method may further include:

[0094] The time point of the renderer after the current offset data is determined as the preset time point.

[0095] It can be understood that a time point between the current time of the renderer and the moment of obtaining the current position data can be selected as the preset time point. In an embodiment of the present invention, the time point of the renderer after determining the current offset data can be selected as the preset time point.

[0096] In this way, the difference between the current time of the renderer and the time point of the renderer after the current offset data is determined is calculated, and then the ratio of the difference to the preset interpolation duration is used as the current offset coefficient to update the current data to be rendered.

[0097] In this way, the preset time point is determined by the above method, so that the determined difference can better reflect the time interval between the current offset data and the rendering of the renderer, which is conducive to determining the rendering coordinates of the current data to be rendered together with the current offset data, so that the rendering coordinates of the current data to be rendered are rendered. The display picture obtained by rendering is closer to the real picture at the current time.

[0098] In order to determine the current offset data, in an optional embodiment, the current data to be rendered is a coordinate structure defined by the renderer, wherein the coordinate structure is a data structure used to represent a point or position in space, usually containing multiple components, each component corresponding to the value of a coordinate axis. For example, the coordinate structure may include a coordinate component of the X axis, a coordinate component of the Y axis, and a coordinate component of the Z axis; the data stored therein is stored as floating point numbers. Accordingly, based on the current data to be rendered and the next data to be rendered, determining the current offset data may include:

[0099] Determine the filter structure corresponding to the current rendering data;

[0100] Filter the filter structure corresponding to the current data to be rendered to obtain the filtered coordinates corresponding to the current data to be rendered;

[0101] The current offset data is determined according to the filtered coordinates corresponding to the current data to be rendered and the filtered coordinates corresponding to the next data to be rendered.

[0102] It can be understood that the current data to be rendered is obtained through the conversion of S103, and the current data to be rendered is a coordinate structure defined by the renderer, that is, the current data to be rendered is stored in the form of a coordinate structure. In order to denoise the current data to be rendered, in an embodiment of the present invention, based on the current data to be rendered stored in the coordinate structure, a corresponding filtering structure can be determined, wherein the filtering structure includes: a coordinate structure and a weight value corresponding to the coordinate structure.

[0103] That is to say, before filtering, the weight value corresponding to each coordinate structure is first determined. Here, the filter structure can also be stored in a first-in-first-out queue during filtering. For example, if the length of the first-in-first-out queue is 4, then 4 filter structures can be stored in the queue. The weight value in the filter structure at each position can be pre-set. For example, the weight values ​​of the queue from the head to the tail can be W1, W2, W3 and W4.

[0104] After obtaining the filter structure corresponding to the current data to be rendered, the filter structure corresponding to the current data to be rendered can be filtered. Here, a weighted average sliding window filtering algorithm can be used. Considering the specific application scenario, a Kalman filtering algorithm or a Smith filtering algorithm can also be used. Here, the embodiment of the present invention does not specifically limit this.

[0105] The above filtering method can be used to obtain the filtered coordinates corresponding to the current data to be rendered, and the current offset data can be obtained by subtracting the filtered coordinates corresponding to the next data to be rendered after the current data to be rendered.

[0106] In this way, the current data to be rendered is stored in a coordinate structure, converted into a filter structure, and filtered, so that the current offset data can be obtained based on the filtered coordinates, providing data for determining the rendering coordinates of the current data to be rendered.

[0107] Furthermore, in order to obtain the filtered coordinates corresponding to the current data to be rendered, in an optional embodiment, filtering the filter structure corresponding to the current data to be rendered to obtain the filtered coordinates corresponding to the current data to be rendered may include:

[0108] The filter structure corresponding to the current data to be rendered and the filter structures corresponding to the next N data to be rendered are filtered to obtain filtered coordinates corresponding to the current data to be rendered.

[0109] It is understandable that filtering can be performed on the filter structure corresponding to the current data to be rendered and the filter structures corresponding to the N data to be rendered following the current data to be rendered, and the resulting filtering results can be used as the filtered coordinates corresponding to the current data to be rendered. Where N is a positive integer greater than or equal to 1. In other words, filtering can be performed on the filter structure corresponding to the current data to be rendered and the filter structures corresponding to the N data to be rendered following the current data to be rendered, and a weighted average sliding window filtering algorithm can be used to obtain the filtered coordinates corresponding to the current data to be rendered.

[0110] Here, a fixed-size first-in-first-out queue can be used to store the filter structure corresponding to the current data to be rendered. For example, the length of the queue is 4. From the head to the tail of the queue, they are: the filter structure corresponding to the current data to be rendered, the filter structure corresponding to the next data to be rendered under the current data to be rendered, the filter structure corresponding to the next data to be rendered under the current data to be rendered, and the filter structure corresponding to the next data to be rendered under the current data to be rendered. After obtaining these four filter structures, the coordinate structure can be weighted and summed based on the weight value in the filter structure to obtain the filtering result, which is the filtered coordinate corresponding to the current data to be rendered.

[0111] In this way, filtering is performed through the filtering structure corresponding to the current data to be rendered and the filtering structure corresponding to the next N data to be rendered, so that the filtered coordinates corresponding to the current data to be rendered are associated with the filtering structures corresponding to the next N data to be rendered, thereby improving the continuity of the display screen through filtering.

[0112] In order to determine the rendering coordinates of the current data to be rendered using the current offset data and the current offset coefficient, in an optional embodiment, S104 may include:

[0113] Using the filtered coordinates, current offset data, and current offset coefficient corresponding to the current data to be rendered, an interpolation frame structure corresponding to the current data to be rendered is formed;

[0114] Draw the rendering coordinates of the current data to be rendered according to the interpolation frame structure corresponding to the current data to be rendered.

[0115] It can be understood that in drawing rendering coordinates, it is necessary to first determine the interpolation structure corresponding to the current data to be rendered. Here, the interpolation structure may include: the filtered coordinates corresponding to the current data to be rendered, the current offset data and the current offset coefficient; the interpolation structure may also include: the filtered coordinates corresponding to the current data to be rendered, the filtered coordinates corresponding to the current data to be rendered, the filtered coordinates corresponding to the next data to be rendered, the current time of the renderer, the preset time point and the interpolation duration; wherein, the current offset data can be determined based on the filtered coordinates corresponding to the current data to be rendered and the filtered coordinates corresponding to the next data to be rendered; the current offset coefficient can be determined based on the current time of the renderer, the preset time point and the interpolation duration.

[0116] After determining the interpolation frame structure corresponding to the current data to be rendered, the rendering coordinates of the current data to be rendered can be drawn according to the interpolation frame structure. Here, the rendering coordinates of the current data to be rendered can be drawn by a renderer.

[0117] In this way, by determining the interpolation structure of the current data to be rendered as described above, the renderer can draw the rendering coordinates of the current data to be rendered based on the interpolation structure. The elements in the interpolation structure are used to correct the filtered coordinates corresponding to the current data to be rendered, so that the rendering coordinates of the current data to be rendered can be drawn by the renderer with the data offset increased, and the rendered picture is closer to the real picture.

[0118] Furthermore, in order to draw the rendering coordinates of the data to be rendered, in an optional embodiment, drawing the rendering coordinates of the data to be rendered according to the interpolation frame structure corresponding to the data to be rendered may include:

[0119] Determine the product of the current offset coefficient and the current offset data;

[0120] The sum of the product and the filtered coordinates corresponding to the current data to be rendered is determined as the rendering coordinates of the current data to be rendered.

[0121] It can be understood that the current offset coefficient can be multiplied by the current offset data first, and then the product can be added to the filtered coordinates corresponding to the current data to be rendered. In this way, the filtered coordinates corresponding to the current data to be rendered can be corrected and the rendering coordinates of the current data to be rendered can be obtained.

[0122] It can be seen that in the embodiment of the present invention, the rendering coordinates of the current data to be rendered are obtained by correcting the filtered coordinates corresponding to the current data to be rendered, so that the rendering coordinates of the current data to be rendered are drawn based on the data collected by the components with the data offset added, thereby improving the smoothness of the picture and making the picture closer to the real picture.

[0123] The scene reconstruction method described in one or more of the above embodiments is described below with examples.

[0124] This example proposes a new motion smoothing display method for environmental reconstruction elements. It uses filtering and interpolation algorithms to process low-frequency position perception data, providing an approximate display effect without relying on high-specification sensor hardware.

[0125] The core system of this method consists of three modules: data receiving module, data filtering module, and data interpolation module.

[0126] The data receiving module receives and processes the real-time position sensing data of the scene reconstruction elements, and transmits the processed 3D coordinate data to the data filtering module. Here, the processing in the data receiving module may include:

[0127] Converts the original scene reconstruction element position-aware data into a 3D coordinate format used by the rendering system, where the 3D coordinate format used by the rendering system can include length units and a coordinate system. Stores the time when the position-aware data is updated to the data receiving module, and calculates the data update interval based on the current system time and the time when the position-aware data was last received.

[0128] Data filtering module: Obtains and stores 3D coordinate data from the data receiving module, performs filtering, and then passes the filtered 3D coordinate data to the data interpolation module. The filtering algorithm uses a weighted average sliding window algorithm (other filtering algorithms such as Kalman filtering and Smith filtering can also be used considering specific application scenarios). This algorithm uses a fixed-size first-in-first-out queue to store 3D coordinate data, and old data is automatically dequeued when new data is added to the queue. The 3D coordinate data output by the data filtering module to the next module is equal to the weighted average result of the data queue. This can be expressed using the following formula:

[0129] (1)

[0130] Among them, k-1 is the queue (window) size, the sum of the weights is equal to 1; i represents the i-th 3D coordinate in the queue. This module aims to suppress the variation amplitude of low-frequency position perception data.

[0131] The Data Interpolation Module obtains and stores 3D coordinate data from the Data Filtering Module. When the rendering system draws a frame, it retrieves intermediate 3D coordinate data generated in real time based on the current frame time from this module. This module uses a first-in, first-out queue of length 2 to store 3D coordinate data. Old data is automatically dequeued as new data is added. The data at the end of the queue is designated as "frame target coordinates," and the data at the beginning of the queue is designated as "frame start coordinates."

[0132] When 3D coordinate data is passed to the data interpolation module, the following operations are performed:

[0133] After the queue is updated, the difference between the frame start 3D coordinate and the frame target 3D position coordinate in 3D space is calculated and stored. This difference is labeled "frame coordinate offset". The frame coordinate offset is calculated using the following formula:

[0134] Frame coordinate offset = frame target coordinate - frame start target (2)

[0135] Get the current system time from the system and store it. This time is marked as "frame start time".

[0136] The data update interval is obtained from the data receiving module and stored. The interval is marked as "interpolation duration".

[0137] Additionally, the renderer performs the following operations when obtaining coordinates from the data interpolation module:

[0138] Get the current time from the system.

[0139] The rendering 3D coordinates are calculated by the following formula and returned to the renderer:

[0140] Drawing coordinates = frame start coordinates + frame coordinate offset × ((current system time - frame start time) / interpolation duration) (3)

[0141] In summary, this example combines the filtering algorithm and the frame interpolation algorithm to ensure similar display effects while effectively reducing the requirements of the vehicle platform for sensor hardware, thereby improving the economic benefits of the entire vehicle.

[0142] Figure 2 A hardware schematic diagram of an optional execution scene reconstruction method provided by an embodiment of the present invention Figure 1 ,like Figure 2 As shown, it includes: a millimeter wave radar 21, a laser radar 22, a visual camera 23, a software platform middleware 24, a data receiving module 25, a data filtering module 26 and a data interpolation module 27.

[0143] In this system, the positional data generated by onboard sensors (which may include millimeter-wave radar 21, lidar 22, and visual cameras 23) is pushed to the system's data receiving module 25 via the software platform middleware 24 according to sensor-specific cycles. The positional data transmitted to the data receiving module 25 is stored in its original sensor-defined format. The data receiving module 25 converts the incoming data into 3D coordinates defined by the renderer, directly using the renderer-defined coordinate structure. Simultaneously, the data receiving module 25 calculates and records the corresponding data update time and period.

[0144] Figure 3 A hardware schematic diagram of an optional execution scene reconstruction method provided by an embodiment of the present invention Figure 2 ,like Figure 3 As shown, Figure 2 In comparison, a system timer 281 is shown in the renderer 28, and the data receiving module 25 also defines an application programming interface (API) for the data interpolation module 27 to query the perception data update period to output the frame start coordinates, frame coordinate offset, frame start time and interpolation duration to the renderer 28.

[0145] Figure 4 A hardware schematic diagram of an optional execution scene reconstruction method provided by an embodiment of the present invention Figure 3 ,like Figure 4The figure shows the filtering algorithm used by data filtering module 26. Data filtering module 26 obtains the 3D coordinates of a scene reconstruction element from data receiving module 25, which are the new coordinates. Depending on whether filtering information for this element has been previously stored, it chooses to create or update the corresponding filter structure (e.g., coordinate 0, coordinate 1, coordinate 2, ..., coordinate k-1). (If a new structure is created, the coordinate queue is completely filled with the current 3D coordinates.) For example, a filter window (queue) size of 4 and weights of (0.1, 0.15, 0.25, 0.5) can be used.

[0146] The output coordinates are calculated using the following formula:

[0147] Output coordinate = weight 0 × coordinate 0 + weight 1 × coordinate 1 + ... + weight k-1 × coordinate k-1 (4)

[0148] The data filtering module 26 updates the coordinate queue, calculates the weighted average result, obtains a filtered 3D coordinate, and sends the coordinate to the data interpolation module 27. The coordinate is not stored in the filter structure.

[0149] Figure 5 A hardware schematic diagram of an optional execution scene reconstruction method provided by an embodiment of the present invention Figure 4 ,like Figure 5 As shown, Figure 2 Compared with the above, the data determined by the data interpolation module 27 is shown. The data interpolation module 27 receives the 3D coordinates (equivalent to Figure 5 The new coordinates in the data filter module 26 are equivalent to the output coordinates of the data filter module 26). Depending on whether the interpolation information of the element has been stored before, the interpolation structure corresponding to the element is created or updated. (If a new structure is created, the coordinate queue is completely filled with the current 3D coordinates.)

[0150] After the data interpolation module 27 updates the coordinate queue:

[0151] 1) Query the sensor data update period from the data receiving module 25 to fill the interpolation frame duration field in the interpolation frame structure.

[0152] 2) Get the current time to fill the frame start time field in the interpolation frame structure.

[0153] 3) Calculate the offset from the frame start coordinate and the frame target coordinate to fill the frame coordinate offset field in the interpolation structure.

[0154] The data interpolation module 27 provides an API for the renderer to query the drawing coordinates, which are calculated by formula (3).

[0155] Thus, this system has improved the smoothness of the motion of environmental reconstruction elements through data filtering and interpolation. Furthermore, testing on the vehicle platform of this example, using sensors of the same specifications, has shown that the frame rate of the environmental reconstruction element motion animation can be increased from 15 frames per second to 25 frames per second, with only a slight increase in central processing unit (CPU) power consumption, remaining essentially the same. This demonstrates the effectiveness of this example.

[0156] An embodiment of the present invention provides a scene reconstruction method, which obtains the current position data of the target object collected and converts it to obtain the current data to be rendered, and then uses the current offset data determined by the current data to be rendered and the subsequent data to be rendered, and uses the current offset data and the current offset coefficient to determine the rendering coordinates of the current data to be rendered, so that the updated rendering coordinates of the current data to be rendered are related to the position data obtained twice before and after, so that the rendering coordinates of the current data to be rendered are closer to the real picture when rendered and displayed, and the current offset coefficient is determined based on the difference between the current time of the renderer and the preset time point, wherein the preset time point is: the time point between the acquisition time of the current position data and the current time of the renderer, so that the current offset coefficient is related to the difference between the current time of the renderer and the time point between the acquisition time of the current position data and the current time of the renderer, so that the rendering coordinates of the current data to be rendered are drawn based on the data collected by the component with the data offset added, so that the rendered picture is closer to the real picture, further improving the smoothness of the displayed picture.

[0157] Based on the same inventive concept as the above embodiments, an embodiment of the present invention provides a scene reconstruction device, Figure 6 A schematic structural diagram of an optional scene reconstruction device provided by an embodiment of the present invention, such as Figure 6 As shown, the scene reconstruction device 600 may include:

[0158] An acquisition module 61 is used to acquire the current location data of the target object;

[0159] The conversion module 62 is used to convert the current position data to obtain the converted current data to be rendered;

[0160] A determination module 63 is configured to determine current offset data based on the current data to be rendered and the previous data to be rendered; wherein the current offset coefficient is determined based on the difference between the current time of the renderer and a preset time point; the preset time point is a time point between the time when the current position data is acquired and the current time of the renderer;

[0161] An updating module 64 is configured to determine the rendering coordinates of the current data to be rendered based on the current data to be rendered, the current offset data, and the current offset coefficient;

[0162] The scene reconstruction module 65 is used to render the rendering coordinates of the current data to be rendered through a renderer to obtain a display image, so as to realize scene reconstruction of the target object.

[0163] In an optional embodiment, the current data to be rendered is a coordinate structure defined by the renderer. Accordingly, the determination module 63 is used to: determine the filter structure corresponding to the current rendering data; the filter structure includes: a coordinate structure and a weight value corresponding to the coordinate structure; filter the filter structure corresponding to the current data to be rendered to obtain the filtered coordinates corresponding to the current data to be rendered; determine the current offset data based on the filtered coordinates corresponding to the current data to be rendered and the filtered coordinates corresponding to the next data to be rendered.

[0164] In an optional embodiment, the determination module 63 filters the filter structure corresponding to the current data to be rendered to obtain the filtered coordinates corresponding to the current data to be rendered, including: filtering the filter structure corresponding to the current data to be rendered and the filter structures corresponding to the next N data to be rendered to obtain the filtered coordinates corresponding to the current data to be rendered; wherein N is a positive integer greater than or equal to 1.

[0165] In an optional embodiment, the update module 64 is used to: use the filtered coordinates corresponding to the current data to be rendered, the current offset data and the current offset coefficient to form an interpolation structure corresponding to the current data to be rendered; and draw the rendering coordinates of the current data to be rendered according to the interpolation structure corresponding to the current data to be rendered.

[0166] In an optional embodiment, the update module 64 draws the rendering coordinates of the current data to be rendered based on the interpolation structure corresponding to the current data to be rendered, including: determining the product of the current offset coefficient and the current offset data; and determining the sum of the product and the filtered coordinates corresponding to the current data to be rendered as the rendering coordinates of the current data to be rendered.

[0167] In an optional embodiment, the device is further used to: determine the proportional coefficient between the difference and the preset interpolation duration as the current offset coefficient; or determine the preset sampling interval duration for the vehicle components as the preset interpolation duration.

[0168] In an optional embodiment, the device is also used to: determine the interval time between the acquisition time of the current position data and the acquisition time of the previous position data as a preset interpolation duration; or determine the time point of the renderer after determining the current offset data as a preset time point.

[0169] In actual applications, the above-mentioned acquisition module 61, conversion module 62, determination module 63, update module 64 and scene reconstruction module 65 can be implemented by a processor located on the scene reconstruction device 600, specifically a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA).

[0170] Figure 7 A schematic diagram of the structure of an optional vehicle-mounted device provided in an embodiment of the present invention, such as Figure 7 As shown, an embodiment of the present invention provides a vehicle-mounted device 700, including:

[0171] A processor 71 and a storage medium 72 storing executable instructions of the processor 71, wherein the storage medium 72 relies on the processor 71 to perform operations through a communication bus 73. When the instructions are executed by the processor 71, the scene reconstruction method described in one or more of the above embodiments is executed.

[0172] It should be noted that in actual application, the various components in the vehicle-mounted device 700 are coupled together via the communication bus 73. It is understood that the communication bus 73 is used to achieve connection and communication between these components. In addition to the data bus, the communication bus 73 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Figure 7 Various buses are labeled as communication buses 73.

[0173] An embodiment of the present invention provides a computer storage medium storing executable instructions. When the executable instructions are executed by one or more processors, the processors execute the scene reconstruction method as described in one or more of the above embodiments.

[0174] An embodiment of the present invention provides a computer program product, including a computer program or instructions. When the computer program or instructions are executed by a processor, the steps of the scene reconstruction method described in one or more embodiments are implemented.

[0175] Among them, the computer-readable storage medium can be a ferromagnetic random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic surface storage device, an optical disc, or a compact disc read-only memory (CD-ROM).

[0176] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage and optical storage) containing computer-usable program code.

[0177] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0178] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0179] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0180] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention.

Claims

1. A scene reconstruction method, characterized in that: include: Get the current location data of the target object; Converting the current position data to obtain converted current data to be rendered; Determining current offset data according to the current data to be rendered and the next data to be rendered; Determining rendering coordinates of the current data to be rendered based on the current data to be rendered, the current offset data, and a current offset coefficient; wherein the current offset coefficient is determined based on a difference between a current time of the renderer and a preset time point; the preset time point is a time point between the time when the current position data is acquired and the current time of the renderer; Rendering the rendering coordinates of the current data to be rendered to obtain a display image, so as to achieve scene reconstruction of the target object; The step of determining the rendering coordinates of the current data to be rendered according to the current data to be rendered, the current offset data, and the current offset coefficient includes: forming an interpolation frame structure corresponding to the current data to be rendered by using the filtered coordinates corresponding to the current data to be rendered, the current offset data, and the current offset coefficient; Determine a product of the current offset coefficient and the current offset data; The sum of the product and the filtered coordinates corresponding to the current data to be rendered is determined as the rendering coordinates of the current data to be rendered.

2. The method according to claim 1, characterized in that The current data to be rendered is a coordinate structure defined by the renderer. Accordingly, determining the current offset data according to the current data to be rendered and the next data to be rendered includes: Determine a filter structure corresponding to the current data to be rendered; the filter structure includes: a coordinate structure and a weight value corresponding to the coordinate structure; Filtering the filter structure corresponding to the current data to be rendered to obtain filtered coordinates corresponding to the current data to be rendered; Current offset data is determined according to the filtered coordinates corresponding to the current data to be rendered and the filtered coordinates corresponding to the next data to be rendered.

3. The method according to claim 2, characterized in that The filtering of the filter structure corresponding to the current data to be rendered to obtain filtered coordinates corresponding to the current data to be rendered includes: Filtering the filter structure corresponding to the current data to be rendered and the filter structures corresponding to the next N data to be rendered of the current data to be rendered to obtain filtered coordinates corresponding to the current data to be rendered; Wherein, N is a positive integer greater than or equal to 1.

4. The method according to any one of claims 1 to 3, characterized in that The method further comprises: Determine the proportional coefficient between the difference and the preset interpolation time as the current offset coefficient; or The time point of the renderer after determining the current offset data is determined as the preset time point.

5. The method according to claim 4, characterized in that The method further comprises: Determine the preset sampling interval duration for the components of the vehicle as the preset interpolation duration; or, The interval between the acquisition time of the current position data and the acquisition time of the previous position data is determined as the preset interpolation duration.

6. A scene reconstruction device, characterized in that: include: An acquisition module is used to obtain the current location data of the target object; A conversion module, configured to convert the current position data to obtain converted current data to be rendered; a determination module, configured to determine current offset data based on the current data to be rendered and the previous data to be rendered; an updating module, configured to determine rendering coordinates of the current data to be rendered based on the current data to be rendered, the current offset data, and a current offset coefficient; wherein the current offset coefficient is determined based on a difference between a current time of the renderer and a preset time point; the preset time point is a time point between the time when the current position data is acquired and the current time of the renderer; A scene reconstruction module is used to render the rendering coordinates of the current data to be rendered by a renderer to obtain a display image, so as to realize scene reconstruction of the target object; Wherein, the update module is used to: forming an interpolation frame structure corresponding to the current data to be rendered by using the filtered coordinates corresponding to the current data to be rendered, the current offset data, and the current offset coefficient; Determine a product of the current offset coefficient and the current offset data; The sum of the product and the filtered coordinates corresponding to the current data to be rendered is determined as the rendering coordinates of the current data to be rendered.

7. A vehicle-mounted device, characterized in that: include: A processor and a storage medium storing instructions executable by the processor, wherein the storage medium relies on the processor to perform operations through a communication bus, and when the instructions are executed by the processor, the scene reconstruction method described in any one of claims 1 to 5 is executed.

8. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the scene reconstruction method according to any one of claims 1 to 5 are implemented.

Images