Scene reconstruction method and device, vehicle-mounted equipment and computer program product

Abstract

The embodiment of the invention discloses a scene reconstruction method and device, vehicle-mounted equipment and a computer program product, and the method comprises the steps: obtaining the current position data of a target object, converting the current position data, obtaining the converted current to-be-rendered data, determining the current offset data according to the current to-be-rendered data and the next to-be-rendered data, and carrying out the rendering of the current to-be-rendered data. According to the current to-be-rendered data, the current offset data and a current offset coefficient, the rendering coordinate of the current to-be-rendered data is determined, and the current offset coefficient is determined based on the difference value between the current time of the renderer and a preset time point; the preset time point is a time point between the acquisition moment of the current position data and the current time of the renderer, rendering the rendering coordinates of the current to-be-rendered data to obtain a display picture for display, so as to realize scene reconstruction of the target object. In this way, the smoothness of the display picture obtained by rendering the collected data through the vehicle-mounted equipment is improved.

Description

Technical Field

[0001] The present invention relates to the technology of scene reconstruction for vehicles, and particularly to a scene reconstruction method, device, vehicle-mounted device, and computer program product. Background Art

[0002] Currently, with the continuous development of social economy and technological progress, the cockpit intelligence of new energy passenger vehicles is getting higher and higher, and the in-vehicle infotainment (IVI) screen is no longer limited to simple information display and basic operation control in the traditional sense.

[0003] Among them, environmental reconstruction is an application that uses sensors integrated in the vehicle to continuously sense the environment around the vehicle, perform 3D reconstruction on important environmental elements, and display them on the IVI screen. This application can provide a more interactive and immersive driving experience.

[0004] However, for environmental elements in a relative motion state, to obtain a smooth and accurate real-time display effect, generally, sensors need to provide high-precision and high-frequency position perception data. However, for some mid- to low-end vehicle-mounted platforms, they do not have corresponding high-specification sensor hardware, resulting in poor smoothness of the display screen. Summary of the Invention

[0005] Embodiments of the present invention provide a scene reconstruction method, device, vehicle-mounted device, and computer program product, which can improve the smoothness of the display screen rendered from the data collected by the vehicle-mounted device.

[0006] The technical solution of the present invention is implemented as follows: Embodiments of the present invention provide a scene reconstruction method, including: Obtaining current position data of a target object; Converting the current position data to obtain converted current data to be rendered; Determining current offset data according to the current data to be rendered and the next data to be rendered; Determining the rendering coordinates of the current data to be rendered according to the current data to be rendered, the current offset data, and the current offset coefficient; wherein, the current offset coefficient is determined based on the difference between the current time of the renderer and a preset time point; the preset time point is: a time point between the acquisition time of the current position data and the current time of the renderer; Rendering the rendering coordinates of the current data to be rendered to obtain a display screen, so as to implement scene reconstruction of the target object.

[0007] In this way, by obtaining the current position data of the target object collected and converting it, the current data to be rendered is obtained. Furthermore, the current offset data determined by the current data to be rendered and the next data to be rendered is used, and its relationship with the current offset coefficient is utilized to determine the rendering coordinates of the current data to be rendered, such that the rendering coordinates of the updated current data to be rendered are related to the position data obtained in the previous and next acquisitions. As a result, when the rendering coordinates of the obtained current data to be rendered are displayed after rendering, the displayed image is closer to the real image. Moreover, the current offset coefficient is determined based on the difference between the current time of the renderer and the preset time point, where the preset time point is the time point between the acquisition moment of the current position data and the current time of the renderer. Thus, the current offset coefficient is related to the current time of the renderer and the difference from the time point between the acquisition moment of the current position data and the current time of the renderer, such that the rendering coordinates of the drawn current data to be rendered have an offset added to the data collected by the component, thereby making the rendered image closer to the real image and further enhancing the smoothness of the displayed image.

[0008] Furthermore, the current data to be rendered is a coordinate structure defined by the renderer. Correspondingly, determining the current offset data according to the current data to be rendered and the next data to be rendered includes: Determining the filtering structure corresponding to the current rendering data; the filtering structure includes: a coordinate structure and the weight value corresponding to the coordinate structure; Filtering the filtering structure corresponding to the current data to be rendered to obtain the filtered coordinates corresponding to the current data to be rendered; Determining the current offset data according to the filtered coordinates corresponding to the current data to be rendered and the filtered coordinates corresponding to the next data to be rendered.

[0009] In this way, by storing the current data to be rendered in the coordinate structure, converting it into a filtering structure, and filtering it, the current offset data can be obtained based on the filtered coordinates, providing data for determining the rendering coordinates of the current data to be rendered.

[0010] Furthermore, filtering the filtering structure corresponding to the current data to be rendered to obtain the filtered coordinates corresponding to the current data to be rendered includes: Filtering the filtering structure corresponding to the current data to be rendered and the filtering structures corresponding to the next N data to be rendered of the current data to be rendered to obtain the filtered coordinates corresponding to the current data to be rendered; where N is a positive integer greater than or equal to 1.

[0011] In this way, filtering is performed through the filtering structure corresponding to the currently to-be-rendered data and the filtering structures corresponding to the subsequent N to-be-rendered data of the currently to-be-rendered data, so that the filtered coordinates corresponding to the currently to-be-rendered data obtained are associated with the filtering structures corresponding to the subsequent N to-be-rendered data, and the continuity of the display screen is improved through filtering.

[0012] Further, determining the rendering coordinates of the currently to-be-rendered data according to the currently to-be-rendered data, the current offset data, and the current offset coefficient includes: Using the filtered coordinates corresponding to the currently to-be-rendered data, the current offset data, and the current offset coefficient to form an interpolation structure corresponding to the currently to-be-rendered data; Drawing the rendering coordinates of the currently to-be-rendered data according to the interpolation structure corresponding to the currently to-be-rendered data.

[0013] In this way, by determining the interpolation structure of the currently to-be-rendered data as described above, the renderer can draw the rendering coordinates of the currently to-be-rendered data based on the interpolation structure, and each element in the interpolation structure corrects the filtered coordinates corresponding to the currently to-be-rendered data, so that the renderer can draw the rendering coordinates of the currently to-be-rendered data with an increased data offset, and the rendered screen is closer to the real screen.

[0014] Further, drawing the rendering coordinates of the currently to-be-rendered data according to the interpolation structure corresponding to the currently to-be-rendered data includes: Determining the product of the current offset coefficient and the current offset data; Taking the sum of the product and the filtered coordinates corresponding to the currently to-be-rendered data as the rendering coordinates of the currently to-be-rendered data.

[0015] In this way, in the embodiment of the present invention, by correcting the filtered coordinates corresponding to the currently to-be-rendered data, the rendering coordinates of the currently to-be-rendered data are drawn, so that the rendering coordinates of the currently to-be-rendered data drawn have an increased data offset on the basis of the data collected by the components, thereby making the screen closer to the real screen while improving the smoothness of the screen.

[0016] Further, the method further includes: Taking the proportional coefficient between the difference value and the preset interpolation duration as the current offset coefficient.

[0017] In this way, the current offset coefficient is determined by the ratio of the above difference to the preset interpolation duration, so that the determined current offset coefficient is inversely proportional to the preset interpolation duration. This enables the rendering coordinates of the current data to be rendered to take into account the preset interpolation duration, thereby improving the accuracy of the rendering coordinates of the current data to be rendered.

[0018] Further, the method further includes: Determine the preset sampling interval duration for the components of the vehicle as the preset interpolation duration.

[0019] In this way, by calculating the current offset coefficient in the above manner, the current offset coefficient is inversely proportional to the preset sampling interval duration of the vehicle components. This enables the rendering coordinates of the current data to be rendered to take into account the preset sampling interval duration of the vehicle components, thereby improving the accuracy of the rendering coordinates of the current data to be rendered.

[0020] Further, the method further includes: Determine the time interval between the acquisition time of the current position data and the acquisition time of the previous position data as the preset interpolation duration.

[0021] In this way, by calculating the current offset coefficient in the above manner, the current offset coefficient is inversely proportional to the time interval between the acquisition times of the two consecutive position data. This enables the rendering coordinates of the current data to be rendered to take into account the time interval between the acquisition time of the current position data and the acquisition time of the previous position data, thereby improving the accuracy of the rendering coordinates of the current data to be rendered.

[0022] Further, the method further includes: Determine the time point of the renderer after the current offset data is determined as the preset time point.

[0023] In this way, by determining the preset time point in the above manner, the determined difference can better reflect the time interval between the determined current offset data and the rendering by the renderer, which is beneficial to determining the rendering coordinates of the current data to be rendered together with the current offset data, making the display image obtained by rendering the rendering coordinates of the current data to be rendered closer to the real image at the current time.

[0024] An embodiment of the present invention provides a scene reconstruction device, including: An acquisition module, configured to acquire the current position data of the target object; A conversion module, configured to convert the current position data to obtain the converted current data to be rendered; A determination module, configured to determine current offset data according to the current data to be rendered and the previous data to be rendered; wherein, the current offset coefficient is determined based on the difference between the current time of the renderer and a preset time point; the preset time point is: the time point between the acquisition moment of the current position data and the current time of the renderer; An update module, configured to determine the rendering coordinates of the current data to be rendered according to the current data to be rendered, the current offset data, and the current offset coefficient; A scene reconstruction module, configured to render the rendering coordinates of the current data to be rendered through a renderer to obtain a display screen, so as to implement scene reconstruction of a target object.

[0025] An embodiment of the present invention provides a vehicle-mounted device, including: a processor and a storage medium storing executable instructions of the processor, the storage medium depends on the processor to execute operations through a communication bus, and when the instructions are executed by the processor, the scene reconstruction method described in one or more of the above embodiments is executed.

[0026] An embodiment of the present invention further provides a computer program product, including a computer program or instructions, characterized in that when the computer program or instructions are executed by a processor, the steps of the scene reconstruction method described in one or more of the above embodiments are implemented.

[0027] Advantages of the present invention: (1) By acquiring the current position data of the target object collected and converting it to obtain the current data to be rendered, and then using the current offset data determined by the current data to be rendered and the next data to be rendered, and using it and the current offset coefficient to determine the rendering coordinates of the current data to be rendered, so that the updated rendering coordinates of the current data to be rendered are related to the position data obtained in the previous and next times, and the display screen when the rendering coordinates of the current data to be rendered obtained are rendered is closer to the real picture; (2) The current offset coefficient is determined based on the difference between the current time of the renderer and the preset time point, wherein the preset time point is: the time point between the acquisition moment of the current position data and the current time of the renderer, so that the current offset coefficient is related to the current time of the renderer and the difference between the time point between the acquisition moment of the current position data and the current time of the renderer, so that the rendering coordinates of the current data to be rendered are drawn with an offset of the data on the basis of the data collected by the components, so that the rendered picture is closer to the real picture, and further improves the smoothness of the display screen. Description of the Drawings

[0028] Figure 1 It is a schematic flowchart of an optional scene reconstruction method provided by an embodiment of the present invention; Figure 2 Hardware schematic diagram of an optional execution scenario reconstruction method provided by an embodiment of the present invention Figure 1 ; Figure 3 Hardware schematic diagram of an optional execution scenario reconstruction method provided by an embodiment of the present invention Figure 2 ; Figure 4 Hardware schematic diagram of an optional execution scenario reconstruction method provided by an embodiment of the present invention Figure 3 ; Figure 5 Hardware schematic diagram of an optional execution scenario reconstruction method provided by an embodiment of the present invention Figure 4 ; Figure 6 Structural schematic diagram of an optional scenario reconstruction device provided by an embodiment of the present invention; Figure 7 Structural schematic diagram of an optional vehicle-mounted device provided by an embodiment of the present invention. Detailed implementation manners

[0029] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention.

[0030] In view of the problem that the sampling frequency of the components of the vehicle-mounted device in the related art is low, resulting in an unsmooth display screen during scenario reconstruction, an embodiment of the present invention provides a scenario reconstruction method. Figure 1 Flow schematic diagram of an optional scenario reconstruction method provided by an embodiment of the present invention, as Figure 1 shown, the scenario reconstruction method may include: S101: Obtain the current position data of the target object; For a vehicle, the current position data of the target object can be collected by components on the vehicle body. For example, the components can collect the current position data of the vehicles around the vehicle, and the components can be cameras, sensors, etc. Among them, the components have a fixed sampling frequency. For example, the current position data of the target object is collected every 10 s. In addition, the above current position data is the position perception data of the scenario reconstruction element in the original format.

[0031] Among them, the above target object can be a vehicle or other moving objects, such as a person or an animal. Here, the embodiments of the present invention do not make specific limitations in this regard.

[0032] It should be noted that the above target object(s) can be one or multiple. When there are multiple target objects, the current position data of each target object can be collected, and then the scene reconstruction method provided by the embodiments of the present invention can be executed for the current position data of each target object, so as to obtain the display screen of each target object and perform the display.

[0033] In addition, the in-vehicle device can also record the time when the current position data of the target object is obtained this time, so that the time when the current position data of the target object is obtained each time can be recorded for subsequent determination of the current offset data.

[0034] S102: Convert the current position data to obtain the converted current data to be rendered; After obtaining the current position data of the target object through the above S101, in S102, the current position data can be converted so that the obtained converted current data to be rendered is in the 3D coordinate format used by the renderer, which is beneficial to realizing the rendering and thus obtaining the display screen.

[0035] In this way, for each target object, the conversion of the current position data can be realized to obtain the current data to be rendered for each target object.

[0036] S103: Determine the current offset data according to the current data to be rendered and the next data to be rendered; After obtaining the current data to be rendered of the target object through the above S102, in S103, the in-vehicle device determines the current offset data according to the current data to be rendered and the next data to be rendered. Here, the current data to be rendered can be subtracted from the next data to be rendered to obtain the current offset data. Among them, the next data to be rendered is the data to be rendered obtained by converting the collected position data after the current data to be rendered. The current data to be rendered and the next data to be rendered are data at two time points, and the time point of the current data to be rendered is before the time point of the next data to be rendered. In addition, both the current data to be rendered and the next data to be rendered are obtained by converting the collected position data before and after. Since each data to be rendered is a three-dimensional coordinate, the subtraction of the current data to be rendered from the next data to be rendered here means: subtracting the current three-dimensional coordinate from the next three-dimensional coordinate.

[0037] Here, it should be noted that in the in-vehicle device, for each target object, a first-in-first-out queue with a length of 2 can be used. In this way, the current data to be rendered can be stored at the head of the queue, and the next data to be rendered can be stored at the tail of the queue. In this way, when obtaining the current data to be rendered at the head of the queue, the current offset data can be determined according to the difference between the current data to be rendered and the next data to be rendered at the tail of the queue, so that the current offset data can be determined for each target object.

[0038] S104: Determine the rendering coordinates of the current data to be rendered according to the current data to be rendered, the current offset data, and the current offset coefficient. After obtaining the current offset data of each target object through the above S103, in S104, determine the rendering coordinates of the current data to be rendered according to the current data to be rendered, the current offset data, and the current offset coefficient. Among them, the current offset coefficient is determined based on the difference between the current time of the renderer and a preset time point; that is to say, after obtaining the current data to be rendered and the current offset data, the current offset coefficient can be determined based on the difference between the current time of the renderer and a preset time point. The preset time point is: the time point between the acquisition moment of the current position data and the current time of the renderer.

[0039] It can be seen that after obtaining the current data to be rendered and the current offset data, first determine the difference between the current time of the renderer and the time point between the acquisition moment of the current position data and the current time of the renderer, and then determine the current offset coefficient based on this difference. Here, the corresponding relationship between the difference and the offset coefficient can be used to determine the current offset coefficient, or a preset calculation formula can be used to calculate the current offset coefficient. Here, the embodiments of the present invention do not make specific limitations on this.

[0040] S105: Render the rendering coordinates of the current data to be rendered to obtain a display screen for display, so as to realize the scene reconstruction of the target object.

[0041] After obtaining the rendering coordinates of the current data to be rendered through the above S104, in S105, the renderer can render the rendering coordinates of the current data to be rendered, so that a display screen can be obtained and the display screen can be displayed. In this way, the scene reconstruction of the target object is realized.

[0042] In this way, through the above display method, even when the acquisition frequency of the components is low, it is still possible to determine the display screen of each frame at the display frame rate and perform display. Without replacing the components, the smoothness of the display screen of the in-vehicle device is improved.

[0043] For the above current offset coefficient, in an optional embodiment, the above method may further include: Determine the proportional coefficient between the difference and the preset interpolation duration as the current offset coefficient.

[0044] It can be understood that in order to calculate the current offset coefficient, in the embodiments of the present invention, the proportional coefficient between the difference and the preset interpolation duration can be used as the current offset coefficient.

[0045] Among them, the preset interpolation duration can be a preset value or a value calculated in real time. Here, the embodiments of the present invention do not make specific limitations in this regard.

[0046] In this way, the current offset coefficient is determined by the ratio of the above difference to the preset interpolation duration, so that the determined current offset coefficient is inversely proportional to the preset interpolation duration, and the rendering coordinates of the current data to be rendered can be made to consider the preset interpolation duration, thereby improving the accuracy of the rendering coordinates of the current data to be rendered.

[0047] For the above preset interpolation duration, in an alternative embodiment, the above method may further include: Determine the preset sampling interval duration for the components of the vehicle as the preset interpolation duration.

[0048] It can be understood that there is a preset sampling interval duration corresponding to the components of the vehicle, and this sampling interval duration is equal to the reciprocal of the sampling interval frequency. Then, after obtaining the preset sampling interval duration, the ratio of the above difference to the sampling interval duration can be used to obtain the current offset coefficient for determining the rendering coordinates of the current data to be rendered.

[0049] In this way, by calculating the current offset coefficient in the above manner, the current offset coefficient is inversely proportional to the preset sampling interval duration of the vehicle components, and the rendering coordinates of the current data to be rendered can be made to consider the preset sampling interval duration of the vehicle components, thereby improving the accuracy of the rendering coordinates of the current data to be rendered.

[0050] For the above preset interpolation duration, in an alternative embodiment, the above method may further include: Determine the interval time between the acquisition time of the current position data and the acquisition time of the previous position data as the preset interpolation duration.

[0051] It can be understood that there is a preset sampling interval duration corresponding to the components of the vehicle, but the sampling interval duration cannot reflect the true interval time between two consecutive data. Here, after obtaining the current position data, record the acquisition time of the current position data. In this way, the acquisition times of two consecutive position data can be known, and the interval time between the acquisition time of the current position data and the acquisition time of the previous position data is used as the preset interpolation duration.

[0052] In this way, by calculating the current offset coefficient in the above manner, such that the current offset coefficient is inversely proportional to the time interval between the acquisition times of the data at the previous and next positions, it is possible to make the rendering coordinates of the currently to-be-rendered data take into account the time interval between the acquisition time of the current position data and the acquisition time of the previous position data, thereby improving the accuracy of the rendering coordinates of the currently to-be-rendered data.

[0053] For the above-mentioned preset time point, in an alternative embodiment, the above method may further include: Determine the time point after the current offset data of the renderer as the preset time point.

[0054] It can be understood that a time point can be selected between the current time of the renderer and the acquisition time of the current position data as the preset time point. In the embodiments of the present invention, the time point after determining the current offset data of the renderer can be selected as the preset time point.

[0055] In this way, calculate the difference between the current time of the renderer and the time point after determining the current offset data of the renderer, and then take the ratio of the difference to the preset interpolation duration as the current offset coefficient, which can be used to update the currently to-be-rendered data.

[0056] In this way, by determining the preset time point in the above manner, the determined difference can better reflect the time interval between determining the current offset data and the renderer performing rendering, which is beneficial to determining the rendering coordinates of the currently to-be-rendered data together with the current offset data, making the display screen obtained by rendering the rendering coordinates of the currently to-be-rendered data closer to the real screen at the current time.

[0057] In order to determine the current offset data, in an alternative embodiment, the currently to-be-rendered data is a coordinate structure defined by the renderer. Among them, the coordinate structure is a data structure used to represent points or positions in space, usually including multiple components, and each component corresponds to the value of an axis. For example, the coordinate structure may include the coordinate component of the X axis, the coordinate component of the Y axis, and the coordinate component of the Z axis; the data stored in it is stored as floating-point numbers. Correspondingly, according to the currently to-be-rendered data and the next to-be-rendered data, determining the current offset data may include: Determine the filter structure corresponding to the current rendered data; Filter the filter structure corresponding to the currently to-be-rendered data to obtain the filtered coordinates corresponding to the currently to-be-rendered data; Determine the current offset data according to the filtered coordinates corresponding to the currently to-be-rendered data and the filtered coordinates corresponding to the next to-be-rendered data.

[0058] Understandably, the current data to be rendered is obtained through the conversion in S103. The current data to be rendered is a coordinate structure defined by the renderer. That is to say, the current data to be rendered is stored in the form of a coordinate structure. In order to denoise the current data to be rendered, in the embodiments of the present invention, based on the current data to be rendered stored in the coordinate structure, a corresponding filtering structure can be determined. The filtering structure includes: a coordinate structure and a weight value corresponding to the coordinate structure.

[0059] That is to say, before filtering, the weight value corresponding to each coordinate structure is determined. Here, in the filtering, the filtering structure can also be stored in a first-in, first-out queue. For example, if the length of the first-in, first-out queue is 4, then 4 filtering structures can be stored in the queue. The weight value in the filtering structure at each position can be preset. For example, the weight values from the head to the tail of the queue can be W1, W2, W3, and W4.

[0060] After obtaining the filtering structure corresponding to the current data to be rendered, the filtering structure corresponding to the current data to be rendered can be filtered. Here, a weighted average sliding window filtering algorithm can be used. Considering the specific application scenario, a Kalman filtering algorithm or a Smith filtering algorithm, etc. can also be used. Here, the embodiments of the present invention do not make specific limitations in this regard.

[0061] By using the above filtering method, the filtered coordinates corresponding to the current data to be rendered can be obtained, and the difference can be taken together with the filtered coordinates corresponding to the next data to be rendered of the current data to be rendered to obtain the current offset data.

[0062] In this way, the current data to be rendered is stored in the coordinate structure, converted into the filtering structure, and filtered, so that the current offset data can be obtained based on the filtered coordinates, providing data for determining the rendering coordinates of the current data to be rendered.

[0063] Further, in order to obtain the filtered coordinates corresponding to the current data to be rendered, in an optional embodiment, filtering the filtering structure corresponding to the current data to be rendered to obtain the filtered coordinates corresponding to the current data to be rendered may include: Filtering the filtering structure corresponding to the current data to be rendered and the filtering structures corresponding to the next N data to be rendered of the current data to be rendered to obtain the filtered coordinates corresponding to the current data to be rendered.

[0064] Understandably, filtering can be performed on the filtering structure corresponding to the currently to-be-rendered data and the filtering structures corresponding to the next N to-be-rendered data of the currently to-be-rendered data, and the obtained filtering result is used as the filtered coordinates corresponding to the currently to-be-rendered data. Here, N is a positive integer greater than or equal to 1. That is to say, filtering can be performed on the filtering structure corresponding to the currently to-be-rendered data and the filtering structures corresponding to the next N to-be-rendered data after the currently to-be-rendered data. The weighted average sliding window filtering algorithm can be used to obtain the filtered coordinates corresponding to the currently to-be-rendered data.

[0065] Here, a first-in-first-out queue with a fixed size can be used to store the filtering structure corresponding to the currently to-be-rendered data. For example, the length of the queue is 4. When the head to the tail of the queue are in turn: the filtering structure corresponding to the currently to-be-rendered data, the filtering structure corresponding to the next to-be-rendered data of the currently to-be-rendered data, the filtering structure corresponding to the next-next to-be-rendered data of the currently to-be-rendered data, and the filtering structure corresponding to the next-next-next to-be-rendered data of the currently to-be-rendered data, after obtaining these four filtering structures, based on the weight values in the filtering structures, weighted summation can be performed on the coordinate structures to obtain the filtering result, which is the filtered coordinates corresponding to the currently to-be-rendered data.

[0066] In this way, by filtering the filtering structure corresponding to the currently to-be-rendered data and the filtering structures corresponding to the next N to-be-rendered data of the currently to-be-rendered data, the filtered coordinates corresponding to the currently to-be-rendered data are associated with the filtering structures corresponding to the next N to-be-rendered data, and the continuity of the display screen is improved through filtering.

[0067] In order to determine the rendering coordinates of the currently to-be-rendered data using the current offset data and the current offset coefficient, in an optional embodiment, S104 may include: Using the filtered coordinates corresponding to the currently to-be-rendered data, the current offset data, and the current offset coefficient, form an interpolation structure corresponding to the currently to-be-rendered data; According to the interpolation structure corresponding to the currently to-be-rendered data, draw the rendering coordinates of the currently to-be-rendered data.

[0068] Understandably, in the rendering coordinates, it is necessary to first determine the interpolation structure corresponding to the currently to-be-rendered data. Here, the interpolation structure may include: the filtered coordinates corresponding to the currently to-be-rendered data, the current offset data, and the current offset coefficient; the interpolation structure may also include: the filtered coordinates corresponding to the currently to-be-rendered data, the filtered coordinates corresponding to the currently to-be-rendered data, the filtered coordinates corresponding to the next to-be-rendered data, the current time of the renderer, the preset time point, and the interpolation duration; among them, the current offset data can be determined according to the filtered coordinates corresponding to the currently to-be-rendered data and the filtered coordinates corresponding to the next to-be-rendered data; the current offset coefficient can be determined according to the current time of the renderer, the preset time point, and the interpolation duration.

[0069] After determining the interpolation structure corresponding to the currently to-be-rendered data, the rendering coordinates of the currently to-be-rendered data can be drawn according to the interpolation structure. Here, the renderer can draw the rendering coordinates of the currently to-be-rendered data.

[0070] In this way, by determining the interpolation structure of the currently to-be-rendered data as described above, the renderer can draw the rendering coordinates of the currently to-be-rendered data based on the interpolation structure, and each element in the interpolation structure corrects the filtered coordinates corresponding to the currently to-be-rendered data, so that the renderer can draw the rendering coordinates of the currently to-be-rendered data with an increased data offset, and the rendered picture is closer to the real picture.

[0071] Furthermore, in order to draw the rendering coordinates of the currently to-be-rendered data, in an optional embodiment, according to the interpolation structure corresponding to the currently to-be-rendered data, drawing the rendering coordinates of the currently to-be-rendered data may include: Determine the product of the current offset coefficient and the current offset data; Determine the sum of the product and the filtered coordinates corresponding to the currently to-be-rendered data as the rendering coordinates of the currently to-be-rendered data.

[0072] Understandably, the current offset coefficient and the current offset data can be multiplied first, and then the product is added to the filtered coordinates corresponding to the currently to-be-rendered data. In this way, the correction of the filtered coordinates corresponding to the currently to-be-rendered data can be realized, and the rendering coordinates of the currently to-be-rendered data can be drawn.

[0073] It can be seen that in the embodiment of the present invention, by correcting the filtered coordinates corresponding to the currently to-be-rendered data, the rendering coordinates of the currently to-be-rendered data are drawn, so that the rendering coordinates of the currently to-be-rendered data drawn are increased with a data offset on the basis of the data collected by the components, thereby improving the smoothness of the picture and making the picture closer to the real picture while.

[0074] The following is an example to describe the scene reconstruction method described in the above one or more embodiments.

[0075] This example proposes a brand-new method for smooth display of the movement of environmental reconstruction elements. By using filtering algorithms and frame interpolation algorithms to process low-frequency position perception data, it provides an approximate display effect without relying on high-specification sensor hardware.

[0076] The core system of this method consists of three modules: a data reception module, a data filtering module, and a data interpolation module.

[0077] Among them, the data reception module receives and processes the real-time position perception data of the scene reconstruction elements, and the processed 3D coordinate data is transmitted to the data filtering module. Here, the processing in the data reception module can include: Converting the position perception data of the scene reconstruction elements in the original format into the 3D coordinate format used by the rendering system. Among them, the 3D coordinate format used by the rendering system can include length units and coordinate systems. Storing the time when the position perception data is updated to this data reception module, and calculating the data update interval time based on the current system time and the time when the last position perception data was received.

[0078] Data filtering module: Obtains and stores the 3D coordinate data from the data reception module, and after performing filtering processing, transmits the filtered 3D coordinate data to the data interpolation module. Among them, the filtering algorithm uses a weighted average sliding window algorithm (other filtering algorithms such as Kalman filtering and Smith filtering can also be used considering the specific application scenario). This algorithm uses a fixed-size first-in-first-out queue to store the 3D coordinate data, and the old data automatically dequeues when new data enqueues. Among them, the 3D coordinate data output by the data filtering module to the next module is equal to the weighted average result of the data queue. It can be expressed by the following formula: (1) Among them, k - 1 is the size of the queue (window), and the sum of the weights is equal to 1; i represents the i-th 3D coordinate in the queue, and this module aims to suppress the change amplitude of low-frequency position perception data.

[0079] Data interpolation module: Obtains and stores the 3D coordinate data from the data filtering module, and when the rendering system draws a frame, it obtains the intermediate 3D coordinate data generated in real time based on the current frame time from this module. This module uses a first-in-first-out queue with a length of 2 to store the 3D coordinate data, and the old data automatically dequeues when new data enqueues. Among them, the data at the end of the queue is marked as "frame target coordinate", and the data at the head of the queue is marked as "frame start coordinate".

[0080] Among them, when the 3D coordinate data is passed into the data interpolation module, the following operations are performed: After the queue is updated, calculate the difference between the starting 3D coordinates of the frame and the target 3D position coordinates of the frame in 3D space and store it. Among them, this difference is identified as "frame coordinate offset". The frame coordinate offset is calculated using the following formula: Frame coordinate offset = Frame target coordinate - Frame starting target (2) Obtain the current system time from the system and store it. Among them, this time is identified as "frame starting time".

[0081] Obtain the data update interval time from the data receiving module and store it. Among them, this time interval is identified as "frame interpolation duration".

[0082] In addition, when the renderer obtains coordinates from this data interpolation module, the following operations are performed: Obtain the current time from the system.

[0083] The rendered 3D coordinates are calculated by the following formula and returned to the renderer: Drawing coordinate = Frame starting coordinate + Frame coordinate offset × ((Current system time - Frame starting time) / Frame interpolation duration) (3) In summary, this example combines the filtering algorithm and the frame interpolation algorithm. While ensuring an approximate display effect, it can effectively reduce the requirements of the in-vehicle platform for sensor hardware, thereby improving the economic benefits of the entire vehicle.

[0084] Figure 2 Hardware schematic of an optional execution scenario reconstruction method provided by an embodiment of the present invention Figure 1 , such as Figure 2 shown, including: millimeter-wave radar 21, lidar 22, vision camera 23, software platform middleware 24, data receiving module 25, data filtering module 26, and data interpolation module 27.

[0085] In this system, first, the position perception data of the vehicle environment elements generated by in-vehicle sensors (which may include millimeter-wave radar 21, lidar 22, vision camera 23) is pushed to the data receiving module 25 of this system through the software platform middleware 24 according to the specific cycle of the sensors. Among them, the position perception data passed into the data receiving module 25 is stored in the original format defined by the sensors, and the data receiving module 25 converts the incoming data into 3D coordinates defined by the renderer. Here, the coordinate structure defined by the renderer is directly used. At the same time, the data receiving module 25 calculates and records the corresponding data update time and period.

[0086] Figure 3 Hardware schematic of an optional execution scenario reconstruction method provided by an embodiment of the present invention Figure 2 , such as Figure 3 shown, is the same as Figure 2In contrast, a system timer 281 is shown in the renderer 28, and the data receiving module 25 simultaneously defines an Application Programming Interface (API) for the data frame interpolation module 27 to query the perception data update period, so as to output the frame start coordinate, frame coordinate offset, frame start time, and frame interpolation duration to the renderer 28.

[0087] Figure 4 Hardware schematic of an optional execution scenario reconstruction method provided by an embodiment of the present invention Figure 3 , such as Figure 4 As shown, the filtering algorithm used by the data filtering module 26 is shown. The data filtering module 26 obtains the 3D coordinates of a scene reconstruction element from the data receiving module 25, which are the new coordinates. According to whether the filtering information of this element has been stored before, it selects to create or update the corresponding filtering structure of this element (for example, coordinate 0, coordinate 1, coordinate 2,..., coordinate k-1). (If a new structure is created, the coordinate queue is filled with the current 3D coordinates.) Example: It can be set that the size of the filtering window (queue) is 4 and the weights are (0.1, 0.15, 0.25, 0.5).

[0088] The output coordinates are calculated using the following formula: Output coordinate = weight 0 × coordinate 0 + weight 1 × coordinate 1 +... + weight k-1 × coordinate k-1 (4) Among them, after the data filtering module 26 updates the coordinate queue, it calculates the weighted average result to obtain a filtered 3D coordinate, and sends this coordinate to the data frame interpolation module 27. This coordinate is not stored in the filtering structure.

[0089] Figure 5 Hardware schematic of an optional execution scenario reconstruction method provided by an embodiment of the present invention Figure 4 , such as Figure 5 As shown, compared with Figure 2 , the data determined in the data frame interpolation module 27 is shown. The data frame interpolation module 27 receives the 3D coordinates from the data filtering module 26 (equivalent to Figure 5 the new coordinates in, equivalent to the output coordinates of the above data filtering module 26), and selects to create or update the corresponding frame interpolation structure of this element according to whether the frame interpolation information of this element has been stored before. (If a new structure is created, the coordinate queue is filled with the current 3D coordinates.) After the data frame interpolation module 27 updates the coordinate queue: 1) Query the sensing data update period from the data receiving module 25 to fill the frame interpolation duration field in the frame interpolation structure.

[0090] 2) Obtain the current time to fill the frame start time field in the frame interpolation structure.

[0091] 3) Calculate the offset from the frame start coordinates and the frame target coordinates to fill the frame coordinate offset field in the frame interpolation structure.

[0092] The data interpolation module 27 provides an API for the renderer to query the drawing coordinates, which are calculated by formula (3).

[0093] So far, the system has improved the motion smoothness of the environmental reconstruction elements through data filtering and data interpolation. Moreover, after testing on the vehicle-mounted platform of this example, using sensors of the same specification, the motion animation frame rate of the environmental reconstruction elements can be increased from 15 frames / s to 25 frames / s. The comparison of the consumption of the Central Processing Unit (CPU) before and after is only slightly increased, and the comparison of the CPU consumption before and after is basically the same. It can be seen that this example is indeed effective.

[0094] The embodiment of the present invention provides a scene reconstruction method. By obtaining the current position data of the target object collected and converting it to obtain the current data to be rendered, and then using the current offset data determined by the current data to be rendered and the next data to be rendered, and using it and the current offset coefficient to determine the rendering coordinates of the current data to be rendered, so that the rendering coordinates of the updated current data to be rendered are related to the position data obtained twice before and after, and the display screen when the rendering coordinates of the obtained current data to be rendered are displayed through rendering is closer to the real screen. Moreover, based on the difference between the current time of the renderer and the preset time point, the current offset coefficient is determined, where the preset time point is: the time point between the acquisition time of the current position data and the current time of the renderer, so that the current offset coefficient is related to the current time of the renderer and the difference between the time point between the acquisition time of the current position data and the current time of the renderer, so that the rendering coordinates of the current data to be rendered are increased by the offset of the data on the basis of the data collected by the components, so that the rendered screen is closer to the real screen, and further improves the fluency of the display screen.

[0095] Based on the same inventive concept as the foregoing embodiment, the embodiment of the present invention provides a scene reconstruction device. Figure 6 As a schematic structural diagram of an optional scene reconstruction device provided by the embodiment of the present invention, as Figure 6 shown, the scene reconstruction device 600 may include: An acquisition module 61, configured to acquire the current position data of the target object; A conversion module 62, configured to convert the current position data to obtain the converted current data to be rendered; A determination module 63 is configured to determine current offset data according to current data to be rendered and previous data to be rendered; wherein, the current offset coefficient is determined based on the difference between the current time of the renderer and a preset time point; the preset time point is: the time point between the acquisition moment of the current position data and the current time of the renderer; An update module 64 is configured to determine the rendering coordinates of the current data to be rendered according to the current data to be rendered, the current offset data, and the current offset coefficient; A scene reconstruction module 65 is configured to render the rendering coordinates of the current data to be rendered through the renderer to obtain a display screen, so as to implement scene reconstruction of the target object.

[0096] In an optional embodiment, the current data to be rendered is a coordinate structure defined by the renderer. Correspondingly, the determination module 63 is configured to: determine a filtering structure corresponding to the current rendering data; the filtering structure includes: a coordinate structure and a weight value corresponding to the coordinate structure; filter the filtering structure corresponding to the current data to be rendered to obtain the filtered coordinates corresponding to the current data to be rendered; determine the current offset data according to the filtered coordinates corresponding to the current data to be rendered and the filtered coordinates corresponding to the subsequent data to be rendered.

[0097] In an optional embodiment, when the determination module 63 filters the filtering structure corresponding to the current data to be rendered to obtain the filtered coordinates corresponding to the current data to be rendered, it includes: filtering the filtering structure corresponding to the current data to be rendered and the filtering structures corresponding to the subsequent N data to be rendered of the current data to be rendered to obtain the filtered coordinates corresponding to the current data to be rendered; wherein, N is a positive integer greater than or equal to 1.

[0098] In an optional embodiment, the update module 64 is configured to: form an interpolation structure corresponding to the current data to be rendered by using the filtered coordinates corresponding to the current data to be rendered, the current offset data, and the current offset coefficient; draw the rendering coordinates of the current data to be rendered according to the interpolation structure corresponding to the current data to be rendered.

[0099] In an optional embodiment, when the update module 64 draws the rendering coordinates of the current data to be rendered according to the interpolation structure corresponding to the current data to be rendered, it includes: determining the product of the current offset coefficient and the current offset data; determining the sum of the product and the filtered coordinates corresponding to the current data to be rendered as the rendering coordinates of the current data to be rendered.

[0100] In an optional embodiment, the apparatus is further configured to: determine the proportional coefficient between the difference value and a preset interpolation duration as the current offset coefficient; or, determine the preset sampling interval duration preset for the components of the vehicle as the preset interpolation duration.

[0101] In an alternative embodiment, the device is further configured to: determine the time interval between the acquisition time of the current position data and the acquisition time of the previous position data as a preset interpolation frame duration; or, determine the time point of the renderer after the current offset data is determined as a preset time point.

[0102] In practical applications, the above-mentioned acquisition module 61, conversion module 62, determination module 63, update module 64, and scene reconstruction module 65 can be implemented by a processor located on the scene reconstruction device 600, specifically a central processing unit (CPU, Central Processing Unit), a microprocessor (MPU, Microprocessor Unit), a digital signal processor (DSP, Digital Signal Processing), or a field programmable gate array (FPGA, Field Programmable Gate Array), etc.

[0103] Figure 7 A schematic structural diagram of an optional vehicle-mounted device provided by an embodiment of the present invention is shown in Figure 7 As shown, an embodiment of the present invention provides a vehicle-mounted device 700, including: A processor 71 and a storage medium 72 storing executable instructions of the processor 71. The storage medium 72 depends on the processor 71 to perform operations through a communication bus 73. When the instructions are executed by the processor 71, the scene reconstruction method described in the above one or more embodiments is executed.

[0104] It should be noted that in practical applications, each component in the vehicle-mounted device 700 is coupled together through a communication bus 73. It can be understood that the communication bus 73 is used to realize the connection and communication between these components. The communication bus 73 includes not only a data bus but also a power bus, a control bus, and a status signal bus. However, for the sake of clear illustration, in Figure 7 all kinds of buses are labeled as the communication bus 73.

[0105] An embodiment of the present invention provides a computer storage medium storing executable instructions. When the executable instructions are executed by one or more processors, the processor executes the scene reconstruction method described in the above one or more embodiments.

[0106] An embodiment of the present invention provides a computer program product, including a computer program or instructions. When the computer program or instructions are executed by a processor, the steps of the scene reconstruction method described in one or more embodiments are implemented.

[0107] Among them, the computer-readable storage medium can be a ferromagnetic random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM), etc.

[0108] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories and optical memories, etc.) that contain computer-usable program code.

[0109] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as the combination of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for realizing the functions specified in Figure 1 one or more flows Figure 1 or a combination of multiple flows and / or blocks

[0110] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device realizes the functions specified in Figure 1 one or more flows Figure 1 or a combination of multiple flows and / or blocks

[0111] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus, so that a series of operation steps are performed on the computer or other programmable apparatus to generate a computer-implemented process, thereby providing instructions for implementing the functions specified in one process or a plurality of processes and / or boxes Figure 1 one process or a plurality of processes and / or boxes Figure 1 steps for implementing the functions specified in one box or a plurality of boxes.

[0112] As mentioned above, it is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention.

Claims

1. A method for scene reconstruction, characterized in that Including: Obtain the current position data of the target object; Convert the current position data to obtain the converted current data to be rendered; Determine the current offset data according to the current data to be rendered and the next data to be rendered; Determine the rendering coordinates of the current data to be rendered according to the current data to be rendered, the current offset data, and the current offset coefficient; wherein, the current offset coefficient is determined based on the difference between the current time of the renderer and a preset time point; the preset time point is: the time point between the acquisition moment of the current position data and the current time of the renderer; Render the rendering coordinates of the current data to be rendered to obtain a display screen, so as to realize the scene reconstruction of the target object.

2. The method according to claim 1, characterized in that, The current data to be rendered is a coordinate structure defined by the renderer. Correspondingly, the step of determining the current offset data according to the current data to be rendered and the next data to be rendered includes: Determine the filter structure corresponding to the current rendering data; the filter structure includes: a coordinate structure and a weight value corresponding to the coordinate structure; Filter the filter structure corresponding to the current data to be rendered to obtain the filtered coordinates corresponding to the current data to be rendered; Determine the current offset data according to the filtered coordinates corresponding to the current data to be rendered and the filtered coordinates corresponding to the next data to be rendered.

3. The method according to claim 2, characterized in that, The step of filtering the filter structure corresponding to the current data to be rendered to obtain the filtered coordinates corresponding to the current data to be rendered includes: Filter the filter structure corresponding to the current data to be rendered and the filter structures corresponding to the next N data to be rendered of the current data to be rendered to obtain the filtered coordinates corresponding to the current data to be rendered; Wherein, N is a positive integer greater than or equal to 1.

4. The method according to claim 2, wherein The step of determining the rendering coordinates of the current data to be rendered according to the current data to be rendered, the current offset data, and the current offset coefficient includes: Use the filtered coordinates corresponding to the current data to be rendered, the current offset data, and the current offset coefficient to form an interpolation structure corresponding to the current data to be rendered; Draw the rendering coordinates of the current data to be rendered according to the interpolation structure corresponding to the current data to be rendered.

5. The method according to claim 4, characterized in that, The step of drawing the rendering coordinates of the current data to be rendered according to the interpolation structure corresponding to the current data to be rendered includes: Determine the product of the current offset coefficient and the current offset data; Determine the sum of the product and the filtered coordinates corresponding to the current data to be rendered as the rendering coordinates of the current data to be rendered.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: Determine the current offset coefficient as the proportional coefficient between the difference value and a preset interpolation duration; or, Determine the time point of the renderer after determining the current offset data as the preset time point.

7. The method according to any one of claims 1 to 5, characterized in that, The method further includes: Determine the preset sampling interval duration for the components of the vehicle as the preset interpolation duration; or, Determine the interval time between the acquisition moment of the current position data and the acquisition moment of the previous position data as the preset interpolation duration.

8. A scene reconstruction device, characterized in that, Including: An acquisition module, configured to acquire the current position data of a target object; A conversion module, configured to convert the current position data to obtain the current data to be rendered after conversion; A determination module, configured to determine the current offset data according to the current data to be rendered and the previous data to be rendered; wherein, the current offset coefficient is determined based on the difference between the current time of the renderer and a preset time point; the preset time point is: a time point between the acquisition moment of the current position data and the current time of the renderer; An update module, configured to determine the rendering coordinates of the current data to be rendered according to the current data to be rendered, the current offset data and the current offset coefficient; A scene reconstruction module, configured to render the rendering coordinates of the current data to be rendered through a renderer to obtain a display screen, so as to implement the scene reconstruction of the target object.

9. A vehicle-mounted device, characterized in that, Including: A processor and a storage medium storing instructions executable by the processor, the storage medium depends on the processor to execute operations through a communication bus, and when the instructions are executed by the processor, the scene reconstruction method according to any one of claims 1 to 7 above is executed.

10. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by the processor, the steps of the scene reconstruction method according to any one of claims 1 to 7 are implemented.

Images