Image rendering method and device, electronic equipment, storage medium and program product

Abstract

The present disclosure relates to an image rendering method and device, electronic equipment, storage medium and program product, relates to the application of image processing technology in the field of vehicles, and the method comprises: acquiring first point cloud data detected by a mobile device at a current time; acquiring second point cloud data detected by the mobile device at a historical time; wherein the historical time refers to a time before the current time; mapping the second point cloud data to a position coordinate system of the mobile device at the current time to determine historical point cloud data; fusing the first point cloud data and the historical point cloud data to obtain to-be-rendered point cloud data; and performing image rendering according to the to-be-rendered point cloud data. The image rendered by the to-be-rendered point cloud data can present more complete information to the user, and the rendering effect and user experience are improved.

Description

Technical Field

[0001] This disclosure relates to the application of image processing technology in the vehicle field, and in particular to image rendering methods, apparatus, electronic devices, storage media, and program products. Background Technology

[0002] In related technologies, mobile devices (such as smart cars) render images based on point cloud data collected by sensors while in motion, in order to dynamically present rendered images of the scene surrounding the mobile device to the user. However, the rendering effect of this method is poor. Summary of the Invention

[0003] To overcome the problems existing in related technologies, this disclosure provides an image rendering method, apparatus, electronic device, storage medium, and program product.

[0004] According to a first aspect of the present disclosure, an image rendering method is provided, the method comprising: acquiring first point cloud data detected by a mobile device at a current time; acquiring second point cloud data detected by the mobile device at a historical time; wherein the historical time refers to a time before the current time; mapping the second point cloud data to the position coordinate system of the mobile device at the current time to determine the historical point cloud data; fusing the first point cloud data and the historical point cloud data to obtain point cloud data to be rendered; and performing image rendering based on the point cloud data to be rendered.

[0005] In this embodiment, first point cloud data detected by the mobile device at the current moment is acquired; second point cloud data detected by the mobile device at a historical moment is acquired; wherein, a historical moment refers to a moment before the current moment; the second point cloud data is mapped to the position coordinate system of the mobile device at the current moment to determine historical point cloud data; the first point cloud data and the historical point cloud data are fused to obtain point cloud data to be rendered; and image rendering is performed based on the point cloud data to be rendered. The point cloud data to be rendered is obtained by fusing historical point cloud data and the first point cloud data detected at the current moment. The historical point cloud data is determined based on the second point cloud data detected at a historical moment before the current moment. Compared with the first point cloud data, the point cloud data to be rendered includes more information. Therefore, the image rendered using the point cloud data to be rendered can present more complete information to the user, especially increasing the probability of presenting backward information of the mobile device to the user, thereby effectively improving the rendering effect and user experience.

[0006] In some possible implementations, the second point cloud data includes at least backward data, which is point cloud data of objects detected at historical moments and located behind the mobile device's direction of motion at the current moment. Since the second point cloud data includes at least backward data, subsequent historical point cloud data is determined based on this second point cloud data. After the historical point cloud data is fused with the first point cloud data to obtain the point cloud data to be rendered, the image rendered based on this point cloud data can provide backward information to the user, thereby effectively improving the rendering effect and user experience.

[0007] In some possible implementations, the spatial range described by the position coordinate system includes multiple regions, each corresponding to a different point cloud fusion method. The first point cloud data is fused with historical point cloud data to obtain the point cloud data to be rendered. This includes: for any region, determining the first data within that region and the second data within that region from historical point cloud data; fusing the first and second data according to the fusion method corresponding to the region to obtain fused data; and combining the fused data from each region to obtain the point cloud data to be rendered. Compared to a globally unified fusion method that does not differentiate between regions, this method of fusion by region and using different fusion methods to fuse point cloud data from different regions is more targeted to the point cloud data of each region. This allows for more refined fusion between the first and historical point cloud data, resulting in the point cloud data to be rendered obtained from the combination of fused data from each region that better reflects the real scene information. In other words, the quality of the final point cloud data to be rendered based on this fusion method is higher, thereby improving the rendering effect of the image based on this point cloud data.

[0008] In some possible implementations, the first data and the second data are fused according to the fusion method corresponding to the region to obtain fused data, including: determining the growth rate of the amount of the first data in the region compared to the amount of the second data in the region; if the growth rate is greater than the growth rate threshold corresponding to the region, the second data in the region is replaced with the first data, and the first data is used as the fused data of the region.

[0009] In this embodiment, a replacement fusion method is adopted based on the growth rate of the data volume of the first data compared to the data volume of the second data. Compared with the superposition fusion method, the data volume of the fused point cloud data can be reduced, thereby reducing the bandwidth pressure of subsequent data transmission and the computational load of subsequent rendering.

[0010] In some possible implementations, the spatial range described by the position coordinate system is a three-dimensional spatial range. Determining the growth rate of the amount of first data in the first point cloud data within the region compared to the amount of second data in the historical point cloud data within the region includes: mapping the three-dimensional spatial range to a two-dimensional space on a horizontal plane, and dividing the two-dimensional space into multiple grids; for any grid, determining the growth rate of the amount of first data in the first point cloud data within the grid compared to the amount of second data in the historical point cloud data within the grid.

[0011] In this embodiment, the three-dimensional spatial range corresponding to the position coordinate system is rasterized so that the point cloud data in each grid can be processed separately. Compared with the method of processing the point cloud data as a whole, the method of processing the point cloud data in each grid separately reduces the computational difficulty.

[0012] In some possible implementations, the growth rate threshold for each region is inversely correlated with the distance between the region and the mobile device. Thus, for regions farther from the mobile device, when the growth rate of the first data within that region is smaller than that of the second data, the point cloud data within that region can be fused. This fusion involves updating the point cloud data within that region to the first data. Subsequently, as the mobile device gradually approaches the region, the point cloud data within that region is progressively updated based on the growth rate threshold corresponding to that region, resulting in a smoother visual effect of scene changes as the mobile device moves closer to the region.

[0013] In some possible implementations, the first point cloud data is fused with historical point cloud data to obtain the point cloud data to be rendered. This includes: determining an updated region and a fixed region within the spatial range described by the position coordinate system; fusing first data from the first point cloud data located within the updated region with second data from the historical point cloud data located within the updated region to obtain fused data; and using the fused data and third data from the historical point cloud data located within the fixed region as the point cloud data to be rendered. In this way, changes in the point cloud data in the updated region can be updated in a timely manner, so that scene changes in these spatial regions can be presented to the user promptly.

[0014] In some possible implementations, the first data and the second data are fused according to the fusion method corresponding to the region to obtain fused data, including: determining the amount of the first data in the region and the amount of the second data in the region; when the amount of the first data and the amount of the second data are both less than the quantity threshold, the first data and the second data are superimposed, and the superimposed data is used as the fused data of the region.

[0015] In this implementation, within any region of the spatial range described by the current position coordinate system, the first and second data are only superimposed when both data volumes are relatively small. This enhances the point cloud data without significantly increasing bandwidth pressure. Furthermore, since the point cloud data is collected directly, the fused data obtained by superimposing the first and second data more accurately reflects objects in the real scene compared to point cloud data enhancement using fictitious data, thus improving rendering quality.

[0016] In some possible implementations, image rendering is performed based on the point cloud data to be rendered, including: The point cloud data to be rendered is transmitted to the image rendering module of the mobile device; wherein, when the current moment is not a full transmission moment, incremental point cloud data is transmitted to the image rendering module, and the incremental point cloud data is the data in the point cloud data to be rendered that has changed since the last data transmission; or, when the current moment is a full transmission moment, all the point cloud data to be rendered is transmitted to the image rendering module. The point cloud data to be rendered, transmitted to the image rendering module, is used for this image rendering.

[0017] In this implementation, when the current moment is not a full-scale transmission moment, incremental point cloud data is transmitted to the image rendering module; when the current moment is not a full-scale transmission moment, all point cloud data to be rendered is transmitted to the mobile image rendering module. This method of transmitting incremental point cloud data and periodically transmitting all point cloud data to be rendered can reduce the error between the rendered image and the actual scene caused by only sending incremental point cloud data, while also reducing the bandwidth pressure on the transmission channel.

[0018] In some possible implementations, before rendering the image based on the point cloud data to be rendered, the method further includes: determining the weight of the point cloud data to be rendered corresponding to any object, wherein the weight is positively correlated with the amount of point cloud data to be rendered corresponding to the object and inversely correlated with the image volume of the object; if the weight is less than a weight threshold, increasing the amount of point cloud data of the object so that the weight of the point cloud data to be rendered used to render the image of the object reaches the weight threshold. This increases the saliency of the object, thereby improving the rendering effect of the object.

[0019] According to a second aspect of the present disclosure, an image rendering apparatus is provided, the apparatus comprising: The first acquisition module is configured to acquire the first point cloud data detected by the mobile device at the current moment; The second acquisition module is configured to acquire second point cloud data detected by the mobile device at a historical moment; where historical moment refers to a moment before the current moment. The determination module is configured to map the second point cloud data to the coordinate system of the mobile device's current position at that moment to determine the historical point cloud data; The fusion module is configured to fuse the first point cloud data with historical point cloud data to obtain the point cloud data to be rendered. The rendering module is configured to render images based on the point cloud data to be rendered.

[0020] In some possible implementations, the second point cloud data includes at least backward data, which is point cloud data of objects detected at historical moments and located behind the direction of movement of the mobile device at the current moment.

[0021] In some possible implementations, the spatial range described by the position coordinate system includes multiple regions, and different regions correspond to different point cloud fusion methods. The fusion module is specifically configured as follows: For any given region, determine the first data in the first point cloud data within the region and the second data in the historical point cloud data within the region; The first and second data are merged according to the fusion method corresponding to the region to obtain the merged data; The fused data from each region is combined to obtain the point cloud data to be rendered.

[0022] In some possible implementations, the fusion module is specifically configured as follows: Determine the growth rate of the first data point in the region compared to the historical data point data of the second data point in the region; If the growth rate is greater than the growth rate threshold corresponding to the region, the second data in the region is replaced with the first data, and the first data is used as the fused data of the region.

[0023] In some possible implementations, the spatial range described by the position coordinate system is a three-dimensional spatial range, and for any given region, the fusion module is specifically configured as follows: The three-dimensional space is mapped onto a two-dimensional space on a horizontal plane, and then the two-dimensional space is divided into multiple grids. For any given grid, determine the growth rate of the first data point cloud data within the grid relative to the growth rate of the second data point cloud data within the grid.

[0024] In some possible implementations, the growth rate threshold for each region is inversely correlated with the distance between the region and the mobile device.

[0025] In some possible implementations, the fusion module is specifically configured as follows: Determine the update region and the fixed region within the spatial range described by the position coordinate system; The first point cloud data located within the update area and the second historical point cloud data located within the update area are merged to obtain the merged data. The third data, which is located within a fixed area and includes the fused data and historical point cloud data, is used as the point cloud data to be rendered.

[0026] In some possible implementations, the fusion module is specifically configured as follows: Determine the amount of data in the first data point cloud data within the region and the amount of data in the second data point cloud data within the region; If the data volume of the first data and the data volume of the second data are both less than the quantity threshold, the first data and the second data are superimposed, and the superimposed data is used as the fused data of the region.

[0027] In some possible implementations, the rendering module is specifically configured as follows: The point cloud data to be rendered is transmitted to the image rendering module of the mobile device; wherein, when the current moment is not a full transmission moment, incremental point cloud data is transmitted to the image rendering module, and the incremental point cloud data is the data in the point cloud data to be rendered that has changed since the last data transmission; or, when the current moment is a full transmission moment, all the point cloud data to be rendered is transmitted to the image rendering module. The point cloud data to be rendered, transmitted to the image rendering module, is used for this image rendering.

[0028] In some possible implementations, the apparatus further includes: The confirmation module is configured to determine the weight of the point cloud data to be rendered for any object. The weight is positively correlated with the amount of point cloud data to be rendered for the object and inversely correlated with the image volume of the object. A new module has been added, configured to increase the amount of point cloud data for an object when the weight is less than the weight threshold, so that the weight of the point cloud data to be rendered in the image of the object reaches the weight threshold.

[0029] According to a third aspect of the present disclosure, a vehicle is provided, comprising: processor; Memory used to store processor-executable instructions; The processor is configured as follows: The steps of performing the image rendering method provided in the first aspect of this disclosure.

[0030] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising: processor; Memory used to store processor-executable instructions; The processor is configured as follows: The steps of performing the image rendering method provided in the first aspect of this disclosure.

[0031] According to a fifth aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the image rendering method provided in the first aspect of the present disclosure.

[0032] According to a sixth aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the image rendering method provided in the first aspect of the present disclosure.

[0033] According to a seventh aspect of the present disclosure, a chip is provided, including a processor and an interface, wherein the processor is configured to read instructions to execute the steps of the image rendering method provided in the first aspect of the present disclosure.

[0034] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0035] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0036] Figure 1 This is a schematic diagram of the architecture of a mobile device according to an exemplary embodiment.

[0037] Figure 2 This is a flowchart illustrating an image rendering method according to an exemplary embodiment.

[0038] Figure 3 This is a schematic diagram of backward data according to an exemplary embodiment.

[0039] Figure 4 This is a schematic diagram illustrating a spatial range described by a position coordinate system according to an exemplary embodiment.

[0040] Figure 5 This is a schematic diagram illustrating a spatial range described by a position coordinate system including an updated region and a fixed region, according to an exemplary embodiment.

[0041] Figure 6 This is a rasterized schematic diagram illustrated according to an exemplary embodiment.

[0042] Figure 7 This is a block diagram of an image rendering apparatus according to an exemplary embodiment.

[0043] Figure 8This is a block diagram illustrating a vehicle according to an exemplary embodiment.

[0044] Figure 9 This is a block diagram illustrating a chip system according to an exemplary embodiment.

[0045] Figure 10 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation

[0046] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0047] It should be noted that all actions involving the acquisition of signals, information, or data in this disclosure are carried out in compliance with the relevant data protection laws and policies of the country where the location is situated, and with authorization from the owner of the relevant device.

[0048] When mobile devices (such as smart cars) are in motion, sensors (such as LiDAR) on the device can collect point cloud data of objects around the device (such as pedestrians, vehicles, roads, buildings, animals, plants, etc.) in real time. The mobile device can then perform image rendering based on the point cloud data, and the rendered image can be displayed on the device's screen for the user to view. This allows the user to understand the changes in the surrounding environment as the device moves, thus improving the safety of the mobile device.

[0049] In related technologies, mobile devices typically use the current frame's point cloud data for rendering, resulting in poor rendering quality and user experience.

[0050] In view of this, the present disclosure provides an image rendering method in which part or all of the historical frame point cloud data is fused with the current frame point cloud data. Compared with the current frame point cloud data, the fused point cloud data includes more information. Therefore, the image rendered with such fused point cloud data can present more complete information to the user, especially increasing the probability of presenting the backward information of the mobile device to the user, thereby effectively improving the rendering effect and user experience.

[0051] Backward information can include information about the area behind the mobile device in its direction of motion. In other words, backward information refers to the perceived data of objects located behind the current direction of motion, observed at historical moments during the mobile device's movement. The direction of motion of the mobile device is not fixed to its orientation, but rather refers to the direction corresponding to its actual displacement in space.

[0052] The image rendering method provided in this disclosure can be applied to mobile devices. For example, mobile devices may include, but are not limited to, vehicles, drones, and other devices capable of acquiring point cloud data and performing image rendering.

[0053] For example, such as Figure 1 As shown, the mobile device 10 may include a processing module 102 and an image rendering module 103.

[0054] The processing module 102 can be used to acquire first point cloud data detected by the mobile device at the current moment; acquire second point cloud data detected by the mobile device at a historical moment (where historical moment refers to a moment before the current moment); map the second point cloud data to the position coordinate system of the mobile device at the current moment to determine historical point cloud data; and fuse the first point cloud data and historical point cloud data to obtain point cloud data to be rendered. Afterwards, the processing module 102 can also be used to send the obtained point cloud data to be rendered to the image rendering module 103.

[0055] The image rendering module 103 can be used to render images based on the point cloud data to be rendered.

[0056] Optionally, such as Figure 1 As shown, the mobile device 10 may also include a sensor 101. The sensor 101 can be used to collect point cloud data and send the collected point cloud data to the processing module 102.

[0057] Optionally, such as Figure 1 As shown, the mobile device 10 may also include a display module 104. The display module 104 can be used to display images rendered by the image rendering module 103.

[0058] In this embodiment of the disclosure, the image rendering method provided can be applied to mobile devices. Compared to application to cloud servers, when applied to mobile devices, point cloud data and rendered images do not need to be transmitted between the cloud server and the mobile device, thus saving data transmission costs and improving image rendering efficiency.

[0059] In other scenarios, the image rendering method provided in this disclosure can be applied to a cloud server. For example, a mobile device sends acquired point cloud data to a cloud server, which then performs image rendering using the image rendering method provided in this disclosure. The cloud server can then send the rendered image to the mobile device for display. This eliminates the need for point cloud data processing and image rendering on the mobile device side, saving memory overhead.

[0060] The image rendering method provided in the embodiments of this disclosure will now be described in conjunction with the accompanying drawings.

[0061] Figure 2 This is a flowchart illustrating an image rendering method according to an exemplary embodiment. The execution entity of this method can be the aforementioned cloud server or... Figure 1 The specific execution entity of the mobile device 10 shown in the embodiment can be determined according to the actual scenario. For example... Figure 2 As shown, the method includes steps S201-S205.

[0062] S201. Obtain the first point cloud data detected by the mobile device at the current moment.

[0063] The first point cloud data may include forward point cloud data detected by the mobile device at the current moment, which may include point cloud data of objects in front of the mobile device. For example, the first point cloud data may include point cloud data of objects scanned within a 120-degree sector area in front of the mobile device at the current moment, centered on the front-rear centerline of the mobile device. These objects may include, for example, pedestrians, vehicles, roads, buildings, animals, plants, etc.

[0064] In this embodiment of the disclosure, the first point cloud data detected by the mobile device at the current moment can be obtained in real time during the movement of the mobile device, so that the user can observe the changes in the scene around the mobile device in real time and respond in real time during the movement of the mobile device.

[0065] In this embodiment of the disclosure, after the mobile device pauses its movement, the frequency of acquiring the first point cloud data detected by the mobile device at the current moment can be reduced, so as to save the computing resources of the mobile device while monitoring possible scene changes around the current location of the mobile device.

[0066] S202. Obtain the second point cloud data detected by the mobile device at a historical moment; where historical moment refers to the moment before the current moment.

[0067] It should be understood that the distance between the mobile device's current location and its historical location is less than or equal to a preset distance, so that the point cloud data fused from the initial and historical point cloud data can include more information. This is because if the distance between the mobile device's current location and its historical location is too great, the point cloud data detected by the mobile device at the historical time may not include at least some scene information about the mobile device's surroundings at the current time. Therefore, the fused point cloud data would not provide more useful scene information compared to the initial point cloud data. Thus, the distance between the mobile device's current location and its historical location can be made less than or equal to a preset distance.

[0068] Accordingly, the second point cloud data can be multiple frames of point cloud data detected by the mobile device at consecutive historical moments before the current moment. Alternatively, the second point cloud data can be any one frame of point cloud data from multiple frames of point cloud data detected by the mobile device at consecutive historical moments before the current moment. The second point cloud data can also be point cloud data obtained by fusing multiple frames of point cloud data detected by the mobile device at consecutive historical moments before the current moment. This application does not limit the number of frames in the second point cloud data or whether the second point cloud data is a fused point cloud data, as long as the distance between the location of the mobile device corresponding to the second point cloud data and the location of the mobile device at the current moment is less than or equal to a preset distance.

[0069] S203. Map the second point cloud data to the coordinate system of the mobile device at the current moment to determine the historical point cloud data.

[0070] The current position coordinate system can be understood as a coordinate system established with the current position of the mobile device as the origin.

[0071] In one possible implementation, the second point cloud data is transformed to the current position coordinate system, and the transformation result is determined as historical point cloud data to provide more complete data support for subsequent rendering.

[0072] In another possible implementation, the second point cloud data is transformed to the current position coordinate system, and the point cloud data within a preset range centered on the origin in the transformation result is determined as historical point cloud data, so as to reduce the amount of computation in subsequent fusion processing while providing more effective data support for subsequent rendering.

[0073] S204. Merge the first point cloud data with the historical point cloud data to obtain the point cloud data to be rendered.

[0074] In a possible implementation, the fusion of the first point cloud data with historical point cloud data can be achieved through point cloud data overlay and / or point cloud data replacement. Specific fusion methods will be detailed in subsequent steps and will not be elaborated upon here.

[0075] S205. Render the image based on the point cloud data to be rendered.

[0076] In this embodiment, first point cloud data detected by the mobile device at the current moment is acquired; second point cloud data detected by the mobile device at a historical moment is acquired; wherein, a historical moment refers to a moment before the current moment; the second point cloud data is mapped to the position coordinate system of the mobile device at the current moment to determine historical point cloud data; the first point cloud data and the historical point cloud data are fused to obtain point cloud data to be rendered; and image rendering is performed based on the point cloud data to be rendered. The point cloud data to be rendered is obtained by fusing historical point cloud data and the first point cloud data detected at the current moment. The historical point cloud data is determined based on the second point cloud data detected at a historical moment before the current moment. Compared with the first point cloud data, the point cloud data to be rendered includes more information. Therefore, the image rendered using the point cloud data to be rendered can present more complete information to the user, especially increasing the probability of presenting backward information of the mobile device to the user, thereby effectively improving the rendering effect and user experience.

[0077] As one possible implementation, the second point cloud data includes at least backward data, which is point cloud data of objects detected at historical moments and located behind the direction of movement of the mobile device at the current moment.

[0078] It is understandable that the second point cloud data includes at least backward data, which is the historical point cloud data obtained by mapping the second point cloud data to the location coordinate system of the mobile device at the current moment, and includes at least the mapping result of the backward data to the location coordinate system of the mobile device at the current moment.

[0079] For example, using a mobile device as a vehicle, the process of the vehicle moving forward and backward is used to illustrate backward data. Figure 3 As shown in Figure 'a', in a scenario where the vehicle is moving forward, the vehicle detects point cloud data S1 at time t1. This point cloud data S1 includes the point cloud data corresponding to road sign A1 and tree B1. The vehicle detects point cloud data S2 at time t2, where road sign A1 is to the right front of the vehicle, and tree B1 is to the left rear. Therefore, the backward data at time t2 can include the point cloud data corresponding to tree B1 from point cloud data S1. Point cloud data S1 and point cloud data S2 are adjacent frames, with time t1 preceding time t2. The acute angle of the dashed line represents the front of the vehicle, and the solid arrow represents the vehicle's movement trajectory.

[0080] like Figure 3As shown in b, in a scenario where the vehicle is reversing or backing up along the same route, the vehicle detects point cloud data S1 at time t1. This point cloud data S1 includes the point cloud data corresponding to road sign A2 and the point cloud data corresponding to tree B2. The vehicle detects point cloud data S2 at time t2 and reverses along the same route at the next time point, t3, without turning. The vehicle detects point cloud data S4 at time t4, where road sign A2 is to the right front of the vehicle and tree B2 is to the left rear. Therefore, the backward data at time t4 can include the point cloud data corresponding to tree B1 from point cloud data S1. Here, time t1 is earlier than time t2, time t2 is earlier than time t3, and time t3 is earlier than time t4.

[0081] like Figure 3 As shown in Figure c, in the scenario where a vehicle is reversing at a fork in the road, the vehicle moves on road L1 and detects point cloud data S1 at time t1. This point cloud data S1 includes the point cloud data corresponding to tree B3. The vehicle detects point cloud data S2 at time t2 and then reverses towards road L2 at the next time point, t3, without turning around. At time t4, the vehicle reverses onto road L2 and detects point cloud data S4, with tree B3 behind the vehicle at time t4. Therefore, the backward data at time t4 can include the point cloud data corresponding to tree B3 from point cloud data S1. Here, time t1 is earlier than time t2, time t2 is earlier than time t3, and time t3 is earlier than time t4.

[0082] In this embodiment, the second point cloud data includes at least backward data. Therefore, historical point cloud data is subsequently determined based on the second point cloud data. After the historical point cloud data is fused with the first point cloud data to obtain the point cloud data to be rendered, the image rendered based on the point cloud data to be rendered can provide backward information to the user, thereby effectively improving the rendering effect and user experience.

[0083] As one possible implementation, step S201 may include: acquiring first source point cloud data detected by the mobile device at the current moment; and determining the point cloud data corresponding to the target category object in the first source point cloud data as the first point cloud data.

[0084] In a possible implementation, the mobile device acquires the first source point cloud data detected by the mobile device at the current moment. The first source point cloud data can be understood as the full-frame point cloud data collected by the mobile device's sensors at the current moment. Then, the first source point cloud data is filtered, and the point cloud data corresponding to objects of the selected target category are determined as the first point cloud data. The target category objects can refer to static objects and / or objects of other categories of interest to the user. Static objects can be, for example, trees, roads, houses, or other objects without speed.

[0085] For example, after acquiring the first source point cloud data detected by the mobile device at the current moment, the first source point cloud data is input into a pre-trained target model to obtain the category to which each object belongs in the first source point cloud data output by the target model. The point cloud data corresponding to the objects of the target category is determined as the first point cloud data. For example, if the objects corresponding to the first source point cloud data include people, vehicles, houses, and trees, after inputting the first source point cloud data into the target model, the target model outputs the point cloud data corresponding to the category of people, vehicles, houses, and trees in the first source point cloud data. Then, the point cloud data corresponding to the category of houses and the point cloud data corresponding to the category of trees are used as the first point cloud data.

[0086] The target model mentioned above can be a random forest model, or a neural network model such as a convolutional neural network or a multilayer perceptron.

[0087] Taking a neural network model as an example, this disclosure will describe the process of training to obtain the target model: Data acquisition equipment collects multiple frames of point cloud data under different environments (such as urban areas, rural areas, and indoor environments) and different weather conditions (such as sunny days, rainy days, and foggy days). This yields multiple sets of training samples, each containing point cloud data samples corresponding to different categories of objects within a single frame of point cloud data. For each frame of point cloud data, the point cloud data of different categories of objects within that frame are labeled; for example, the point cloud data of a tree trunk is labeled "tree trunk," and the point cloud data of a house is labeled "house."

[0088] The data acquisition device sends multiple sets of training samples to the training device, which uses these training samples as input and the corresponding labels as output samples to train the neural network model. The model parameters are adjusted based on the difference between the actual output of the neural network model and the output samples. Training ends when the difference between the actual output of the neural network model and the output samples is less than a preset difference; the neural network model at this point is the target model.

[0089] The training device can be a terminal, or other computing devices such as servers or cloud devices. For example, the training device can be a neural network processor, a microprocessor, an application-specific integrated circuit, or one or more integrated circuits used to control the execution of the program of this disclosure. The form of the data acquisition device can be similar to that of the training device described above, and will not be elaborated here.

[0090] In this embodiment, the first point cloud data is taken from the point cloud data corresponding to the target category of objects in the first source point cloud data. This way, during subsequent fusion and rendering, only the point cloud data corresponding to the target category objects is processed. This avoids interference from point cloud data corresponding to objects of other categories, thereby improving the fusion and rendering effect of the point cloud data corresponding to the target category objects. Furthermore, since the first point cloud data is only a portion of the first source point cloud data, processing the first point cloud data alone reduces computational load and improves processing efficiency compared to processing the entire first source point cloud data.

[0091] As one possible implementation, the spatial range described by the position coordinate system includes multiple regions, each corresponding to a different point cloud fusion method. The first point cloud data is fused with historical point cloud data to obtain the point cloud data to be rendered, including: For any given region, determine the first data of the first point cloud data within that region and the second data of the historical point cloud data within that region; fuse the first data and the second data according to the fusion method corresponding to that region to obtain fused data; combine the fused data of each region to obtain the point cloud data to be rendered.

[0092] In this embodiment of the disclosure, the spatial range described by the position coordinate system is actually the spatial range described by the position coordinate system at the current moment. For ease of description, it is described as the spatial range described by the position coordinate system.

[0093] Optionally, the spatial range described by the position coordinate system can be the entire spatial range described by the position coordinate system at the current moment, so as to provide more complete data support for subsequent rendering.

[0094] Optionally, the spatial range described by the position coordinate system can also be a spatial region of a certain size within the entire spatial range described by the position coordinate system at the current moment, including the geographical location of the mobile device. This size is used to determine the size of the spatial range described by the position coordinate system. That is, the spatial range described by the position coordinate system can be the aforementioned preset range centered on the origin, in order to reduce the computational load of subsequent fusion processing while providing more effective data support for subsequent rendering.

[0095] For example, the spatial range described by the position coordinate system can be as follows: Figure 4 As shown, for any frame of point cloud data, the spatial range described by the position coordinate system can be the region enclosed by X[x1, x2], Y[y1, y2], and Z[0, z1] with the center of the mobile device as the origin P, and the coordinates of the origin P can be (0, 0). For example, x1 can be -20 meters, x2 can be 20 meters, y1 can be -40 meters, and y2 can be 60 meters.

[0096] It is understandable that, for any frame of point cloud data, the point cloud data located within the spatial range described by the position coordinate system can be understood as the point cloud data corresponding to the objects located within the spatial range described by the position coordinate system.

[0097] In this implementation, the spatial range described by the position coordinate system includes multiple regions, each corresponding to a different point cloud fusion method. For any given region, the first data of the first point cloud data within that region and the second data of the historical point cloud data within that region are determined. Then, the first and second data are fused according to the fusion method corresponding to the region to obtain fused data. The fused data from each region are combined to obtain the point cloud data to be rendered. Compared to a globally unified fusion method that does not differentiate between regions, this method of fusion by region and using different fusion methods to fuse point cloud data from different regions is more targeted to the point cloud data of each region. It allows for more refined fusion between the first and historical point cloud data, resulting in the point cloud data to be rendered obtained based on the fused data from each region that better reflects the real scene information. In other words, the quality of the final point cloud data to be rendered based on this fusion method is higher, thereby improving the rendering effect of the image based on the point cloud data to be rendered.

[0098] As one possible implementation, the first data and the second data are fused according to the fusion method corresponding to the region to obtain fused data, including: Determine the growth rate of the first data in the first point cloud data within the region compared to the growth rate of the second data in the historical point cloud data within the same region; if the growth rate is greater than the growth rate threshold corresponding to the region, replace the second data in the region with the first data, and use the first data as the fused data of the region.

[0099] The growth rate threshold for any region can be set according to the actual scenario. It should be understood that the growth rate threshold should be set so that the first data is used to replace the second data in all regions not described by the location coordinate system. This is because if the first data is used to replace the second data in all regions described by the location coordinate system, the fused point cloud data becomes only the first point cloud data of the spatial range described by the location coordinate system, without using historical point cloud data, resulting in a lack of backward information in the fused point cloud data. For example, the growth rate threshold for any region can be 10%.

[0100] In a possible implementation, the data volume of the first data and the data volume of the second data are determined within each region of the spatial range described by the location coordinate system. For any region, the growth rate of the data volume of the first data in that region relative to the data volume of the second data is calculated. If the growth rate is greater than the growth rate threshold corresponding to that region, the mobile device has acquired richer information at the current moment. Therefore, the second data in that region is replaced with the first data, and the first data is used as the fused data for that region. If the growth rate is less than or equal to the growth rate threshold corresponding to that region, it indicates that the information acquired by the mobile device at the current moment has changed little or not at all compared to the information acquired by the mobile device at a historical location. In other words, the information acquired by the mobile device at a historical location is richer than the information acquired by the mobile device at the current moment. Therefore, the second data in that region is retained, and the first data in that region is discarded. The second data is used as the fused data for that region.

[0101] For example, the growth rate threshold for each region is 10%. For any region, if the amount of the first data within that region is 120 and the amount of the second data is 100, the increase in the amount of the first data compared to the second data (20) is greater than 10% of the amount of the second data. Therefore, the first data within that region replaces the second data, and the first data becomes the fused data for that region. In other words, after fusion, the point cloud data within that region is the first data. If the amount of the first data within that region is 105 and the amount of the second data is 100, the increase in the amount of the first data compared to the second data (5) is less than 10% of the amount of the second data. Therefore, the second data within that region is retained, and the second data becomes the fused data for that region. In other words, after fusion, the point cloud data within that region is the second data.

[0102] It is understandable that if the growth rate is greater than or equal to the growth rate threshold corresponding to the region, the second data in the region can be replaced with the first data; if the growth rate is less than the growth rate threshold corresponding to the region, the second data in the region can be retained and the first data in the region can be discarded.

[0103] In the above embodiments, a replacement fusion method is adopted based on the growth rate of the data volume of the first data compared to the data volume of the second data. Compared with the superposition fusion method, the data volume of the fused point cloud data can be reduced, thereby reducing the bandwidth pressure of subsequent data transmission and the computational load of subsequent rendering.

[0104] In the above method of merging data based on the growth rate of the first data volume compared to the second data volume, the growth rate threshold for each region can be the same. In other possible implementations, the growth rate thresholds for different regions can also be different.

[0105] As one possible implementation, the spatial range described by the location coordinate system has multiple regions, and the growth rate threshold corresponding to each region is inversely correlated with the distance between the region and the mobile device.

[0106] Generally, the farther the sensor on the mobile device is from the object being scanned, the less point cloud data the sensor collects; conversely, the closer the sensor is to the object, the more point cloud data the sensor collects. Therefore, the growth rate threshold for each region within the spatial range described by the location coordinate system can be inversely correlated with the distance between the region and the mobile device. That is, within the spatial range described by the location coordinate system, the farther a region is from the mobile device, the smaller its growth rate threshold; and the closer a region is to the mobile device, the larger its growth rate threshold. Thus, for regions far from the mobile device, when the growth rate of the first data point within that region is smaller than that of the second data point, the point cloud data within that region can be fused. This fusion involves updating the point cloud data within that region to the first data point. Subsequently, as the mobile device gradually approaches that region, the point cloud data within that region is progressively updated based on the growth rate threshold corresponding to that region, resulting in a smoother visual effect of scene changes as the mobile device moves closer to the region.

[0107] In some embodiments, the different regions mentioned above correspond to different point cloud fusion methods, which may include fusing different regions based on different growth rate thresholds in the above embodiments.

[0108] As one possible implementation, the first data and the second data are fused according to the fusion method corresponding to the region to obtain fused data, including: Determine the amount of the first data in the region and the amount of the second data in the region; if the amount of the first data and the amount of the second data are both less than the quantity threshold, overlay the first data and the second data, and use the overlaid data as the fused data of the region.

[0109] In real-world scenarios, some objects have low surface reflectivity. For such objects, regardless of the mobile device's current or historical location, the amount of point cloud data collected by the mobile device's sensors is relatively small, resulting in poor rendering of these objects in the image rendered based on the fused point cloud data. Therefore, to improve the rendering effect of objects with small point cloud data, a possible implementation is to determine whether the data volume of the first data point in that region and the data volume of the second data point in that region are both less than a threshold. If so, the first and second data points are superimposed, and the superimposed data is used as the fused data for that region.

[0110] In this implementation, within any region of the spatial range described by the position coordinate system, the first and second data are only superimposed when both are relatively small. This enhances the point cloud data without significantly increasing bandwidth pressure. Furthermore, since the point cloud data is collected directly, the fused data obtained by superimposing the first and second data more accurately reflects objects in the real scene compared to point cloud data enhancement using fictitious data, thus improving rendering quality.

[0111] Optionally, before determining the amount of first data in the region and the amount of second data in the region of historical point cloud data, the method further includes: determining the category of the object to which the point cloud data in the region belongs as the target category.

[0112] The target category can refer to the category of static objects and / or other categories that users are interested in. Static objects can be, for example, objects that do not move, such as trees, roads, and houses.

[0113] In a possible implementation, point cloud data can carry category identifiers, which indicate the category of the object to which the point cloud data belongs. For any region within the spatial range described by the position coordinate system, only if the category identifiers carried by the point cloud data within that region indicate that the object to which the point cloud data belongs belongs to the target category, is the step of determining the data volume of the first data and the second data within that region performed. If both the data volumes of the first and second data are less than a quantity threshold, the first and second data are superimposed. That is, this step is performed only on a portion of the spatial range described by the position coordinate system, thus reducing the amount of data calculated. Furthermore, interference from objects of non-target categories can be avoided, thereby improving the data quality of the subsequently superimposed point cloud data to be rendered, and consequently improving the rendering effect based on this point cloud data.

[0114] This disclosure does not limit the specific implementation of point cloud data fusion for objects of non-target categories. For example, the aforementioned substitution fusion method can be used for fusion.

[0115] As one possible implementation, the first point cloud data is fused with historical point cloud data to obtain the point cloud data to be rendered, including: Determine the update region and fixed region within the spatial range described by the position coordinate system; fuse the first data of the first point cloud data located within the update region and the second data of the historical point cloud data located within the update region to obtain fused data; use the fused data and the third data of the historical point cloud data located within the fixed region as the point cloud data to be rendered.

[0116] In real-world scenarios, users may only focus on one or more spatial regions located at a fixed distance from their mobile devices. To ensure timely updates to the point cloud data in these regions and to present these scene changes to the user promptly, these spatial regions can be defined as the update areas within the spatial range described by the position coordinate system, while areas outside the update areas are defined as fixed areas. Then, the first and second data points within the update areas are merged to obtain fused data. Finally, the fused data, along with third data points from historical point cloud data located within the fixed areas, are used as the point cloud data to be rendered.

[0117] For example, such as Figure 5 As shown, the update region can be one or more regions within the spatial range described by the position coordinate system. Figure 5 Taking a spatial range described by a position coordinate system as an example, there are three update regions, with the remaining regions being fixed regions. Each time point cloud data is fused, only the point cloud data within these three update regions is fused separately, resulting in fused data for each of the three update regions. Then, this fused data, along with third data from historical point cloud data located within the fixed regions, is used as the point cloud data to be rendered.

[0118] In one possible implementation, fusing the first and second data within the updated region can include: for any updated region, superimposing the first and second data within the updated region to make the updated region more salient, thereby improving the rendering effect of the updated region.

[0119] In another possible implementation, the fusion of the first and second data within the update region can include: for any update region, if the amount of the first data and the amount of the second data within the update region are both less than the quantity threshold, the first data and the second data within the update region are superimposed, so as to make the salience of the update region stronger without increasing bandwidth pressure as much as possible, thereby making the rendering effect of the update region better.

[0120] If the amount of data in the first data within the updated region is greater than the amount of data in the second data, then the first data in the updated region is used to replace the second data in the updated region. In this way, the amount of data in the fused data obtained by merging the updated regions is slightly larger than the amount of data in the second data within the updated regions, but smaller than the amount of data in the fused data obtained by overlay fusion, thus balancing bandwidth pressure and fusion quality.

[0121] As one possible implementation, the spatial range described by the position coordinate system is a three-dimensional spatial range. Determining the growth rate of the amount of first point cloud data within the region compared to the amount of second historical point cloud data within the region includes: The three-dimensional spatial range is mapped to a two-dimensional space on a horizontal plane, and multiple grids are divided in the two-dimensional space. For any grid, the growth rate of the first data of the first point cloud data within the grid is determined relative to the growth rate of the second data of the historical point cloud data within the grid.

[0122] In a possible implementation, the coordinates of points in point cloud data are typically three-dimensional coordinates. The spatial extent described by the position coordinate system is mapped to a two-dimensional space on a horizontal plane, and this two-dimensional space is divided into multiple grids of equal size. The point cloud data within the spatial extent described by the position coordinate system is then mapped to these grids in the two-dimensional space. Next, the growth rate of the data volume of the first data point in each grid relative to the data volume of the second data point is determined.

[0123] For example, Figure 6 This is a rasterized schematic diagram illustrating an exemplary embodiment. For example... Figure 6 As shown, the spatial range described by the position coordinate system can be mapped to a two-dimensional space on the horizontal plane as an area bounded by X[-20, 20] and Y[-40, 60] with the center of the mobile device as the origin P(0, 0). This two-dimensional space is divided into a grid of 8×20 grids with a side length of 5.

[0124] In this embodiment, the spatial range described by the position coordinate system is rasterized so that the point cloud data in each grid can be processed separately. Compared with processing the point cloud data as a whole, processing the point cloud data in each grid separately reduces the computational difficulty.

[0125] Typically, the process of obtaining the point cloud data to be rendered is performed by the processing module in the mobile device, while the rendering process is performed by the image rendering module. Therefore, the point cloud data to be rendered involves transfer between the processing module and the image rendering module. In some scenarios, the obtained point cloud data can be transferred to the image rendering module so that the subsequently rendered image includes more scene details.

[0126] As one possible implementation, image rendering based on the point cloud data to be rendered includes: The point cloud data to be rendered is transmitted to the image rendering module of the mobile device; wherein, when the current moment is not a full transmission moment, incremental point cloud data is transmitted to the image rendering module, and the incremental point cloud data is the data in the point cloud data to be rendered that has changed since the last data transmission; or, when the current moment is a full transmission moment, all the point cloud data to be rendered is transmitted to the image rendering module. The point cloud data to be rendered, transmitted to the image rendering module, is used for this image rendering.

[0127] In a possible implementation, the point cloud data to be rendered may carry an update identifier, which can be identified and transmitted to the image rendering module of the mobile device for rendering. This disclosure does not limit the specific rendering process of the image rendering module. For example, the image rendering module can render the data transmitted at the current moment into the image rendered at the previous moment. Another example is that the image rendering module can perform the current image rendering based on the data transmitted at the current moment and the data rendered at the previous moment. Yet another example is that the image rendering module can also perform the current image rendering based on the data transmitted at the current moment and point cloud data rendered at multiple historical moments.

[0128] The point cloud data to be rendered can be sent according to a preset period, and the end of each period can be understood as the time of full transmission. For example, the preset period can be 0.5 seconds.

[0129] If only incremental point cloud data is transmitted for rendering each time, the error between the rendered image and the actual scene can easily increase after multiple transmissions. Therefore, the point cloud data to be rendered can be transmitted to the image rendering module periodically. This reduces the error between the rendered image and the actual scene caused by only sending incremental point cloud data, while also reducing the bandwidth pressure on the transmission channel. Specifically, when it is determined that the current moment is not a full transmission moment, incremental point cloud data is transmitted to the image rendering module; when it is determined that the current moment is a full transmission moment, all point cloud data to be rendered is transmitted to the image rendering module of the mobile device.

[0130] For example, point cloud data to be rendered is sent at two full transmission moments: 0 seconds and 0.5 seconds. The next full transmission moment is 1 second. At the current moment of 0.6 seconds, the full transmission moment has not yet arrived, so only the incremental point cloud data of the current moment is transmitted to the image rendering module. The image rendering module can, for example, fuse the point cloud data to be rendered received at 0 seconds and 0.5 seconds, as well as the incremental point cloud data received at 0.6 seconds, to obtain first rendering data. The image rendering module can then perform image rendering based on this first rendering data. At the current moment of 0.7 seconds, the full transmission moment has not yet arrived, so only the incremental point cloud data of the current moment is transmitted to the image rendering module. The image rendering module can, for example, fuse the point cloud data to be rendered received at 0.5 seconds, the incremental point cloud data received at 0.6 seconds, and the incremental point cloud data received at 0.7 seconds, to obtain second rendering data. The image rendering module can then perform image rendering based on this second rendering data.

[0131] Optionally, to avoid the image rendering module being unable to process the data due to excessive data volume, the image rendering module can determine the total amount of rendering data within each preset period. If the total amount of data exceeds the data volume threshold, the earliest point cloud data received within that period will be discarded.

[0132] For example, the point cloud data to be rendered is sent at two full transmission times: 0 seconds and 0.5 seconds. If the current time is 0.7 seconds, and incremental point cloud data has been transmitted to the image rendering module at both 0.6 seconds and 0.7 seconds, and the total data volume of the point cloud data received at 0.5 seconds, the incremental point cloud data received at 0.6 seconds, and the incremental point cloud data received at 0.7 seconds exceeds the data volume threshold, then all or part of the point cloud data received at 0.5 seconds can be discarded. Only the incremental point cloud data received at 0.6 seconds, the fused data received at 0.7 seconds, and the remaining point cloud data from the discarded point cloud data received at 0.5 seconds are fused to obtain the third rendering data. The image rendering module then performs image rendering based on this third rendering data.

[0133] Optionally, to further reduce bandwidth pressure, data compression algorithms can be applied to compress the transmitted point cloud data, minimizing the size of data packets while ensuring information integrity.

[0134] Optionally, an error detection mechanism can be introduced during the transmission of point cloud data to ensure the stability and reliability of data transmission.

[0135] As one possible implementation, the method further includes, before rendering the image based on the point cloud data to be rendered: Determine the weight of the point cloud data to be rendered for any object. The weight is positively correlated with the amount of point cloud data to be rendered for the object and inversely correlated with the image volume of the object. If the weight is less than the weight threshold, increase the amount of point cloud data of the object so that the weight of the point cloud data to be rendered in the image of the object reaches the weight threshold.

[0136] The image volume of an object can be understood as the volume of the 3D image when the object is rendered. For example, the image volume of a tree can be understood as the volume of the 3D tree image when the tree is rendered.

[0137] In a possible implementation, for any object in the point cloud data to be rendered, the image volume of the object is determined based on the corresponding point cloud data. The ratio between the amount of data in the point cloud data corresponding to that object and the image volume of the object is then determined as the weight of the point cloud data corresponding to that object. If the weight is less than a weight threshold, the amount of point cloud data for that object is increased so that the weight of the point cloud data used to render the object's image reaches the weight threshold, thereby increasing the saliency of the object and improving its rendering effect. Increasing the amount of point cloud data for that object can be understood as creating virtual point cloud data for that object within the point cloud data to be rendered, increasing the density of the point cloud data corresponding to that object. Different object categories may have different weight thresholds.

[0138] For example, the point cloud data to be rendered includes point cloud data corresponding to trees and point cloud data corresponding to streetlights. Taking the weight determination process of the point cloud data corresponding to trees as an example: based on the point cloud data corresponding to trees, the image volume of the tree is determined to be approximately 0.1m × 0.1m × 0.5m, and the data volume of the point cloud data corresponding to trees is 100. Then, the weight of the point cloud data corresponding to trees is 100 / (0.1 × 0.1 × 0.5). If the weight threshold for trees is 500 / (0.1 × 0.1 × 0.5), then the density of the point cloud data corresponding to trees is increased so that the weight of the point cloud data corresponding to trees reaches 500 / (0.1 × 0.1 × 0.5).

[0139] It is understandable that the newly added point cloud data can also carry an update identifier. In the above embodiments, when transmitting fused data, the newly added point cloud data can also be transmitted to improve the rendering effect when the data is not transmitted in full.

[0140] Optionally, the point cloud data to be rendered obtained at the current moment can be used as the historical point cloud data for the next current moment.

[0141] Figure 7This is a block diagram illustrating an image rendering apparatus according to an exemplary embodiment. (Refer to...) Figure 7 The image rendering device 70 includes a first acquisition module 701, a second acquisition module 702, a determination module 703, a fusion module 704, and a rendering module 705.

[0142] The first acquisition module 701 is configured to acquire the first point cloud data detected by the mobile device at the current location; The second acquisition module 702 is configured to acquire second point cloud data detected by the mobile device at a historical location; The determination module 703 is configured to determine historical point cloud data within the spatial range described by the location coordinate system of the mobile device based on the second point cloud data. The fusion module 704 is configured to fuse the first point cloud data with historical point cloud data located within the spatial range described by the position coordinate system to obtain the point cloud data to be rendered. Rendering module 705 is configured to render an image based on the point cloud data to be rendered.

[0143] In some possible implementations, the second point cloud data includes at least backward data, which is point cloud data of objects detected at historical moments and located behind the direction of movement of the mobile device at the current moment.

[0144] In some possible implementations, the spatial range described by the position coordinate system includes multiple regions, with different regions corresponding to different point cloud fusion methods. The fusion module 704 is specifically configured as follows: For any given region, determine the first data in the first point cloud data within the region and the second data in the historical point cloud data within the region; The first and second data are merged according to the fusion method corresponding to the region to obtain the merged data; The fused data from each region is combined to obtain the point cloud data to be rendered.

[0145] In some possible implementations, the fusion module 704 is specifically configured as follows: Determine the growth rate of the first data point in the region compared to the historical data point data of the second data point in the region; If the growth rate is greater than the growth rate threshold corresponding to the region, the second data in the region is replaced with the first data, and the first data is used as the fused data of the region.

[0146] In some possible implementations, the spatial range described by the position coordinate system is a three-dimensional spatial range. For any given region, the fusion module 704 is specifically configured as follows: The three-dimensional space is mapped onto a two-dimensional space on a horizontal plane, and then the two-dimensional space is divided into multiple grids. For any given grid, determine the growth rate of the first data point cloud data within the grid relative to the growth rate of the second data point cloud data within the grid.

[0147] In some possible implementations, the growth rate threshold for each region is inversely correlated with the distance between the region and the mobile device.

[0148] In some possible implementations, the fusion module 704 is specifically configured as follows: Determine the update region and the fixed region within the spatial range described by the position coordinate system; The first point cloud data located within the update area and the second historical point cloud data located within the update area are merged to obtain the merged data. The third data, which is located within a fixed area and includes the fused data and historical point cloud data, is used as the point cloud data to be rendered.

[0149] In some possible implementations, the fusion module 704 is specifically configured as follows: Determine the amount of data in the first data point cloud data within the region and the amount of data in the second data point cloud data within the region; If the data volume of the first data and the data volume of the second data are both less than the quantity threshold, the first data and the second data are superimposed, and the superimposed data is used as the fused data of the region.

[0150] In some possible implementations, the rendering module 705 is specifically configured as follows: The point cloud data to be rendered is transmitted to the image rendering module of the mobile device; wherein, when the current moment is not a full transmission moment, incremental point cloud data is transmitted to the image rendering module, and the incremental point cloud data is the data in the point cloud data to be rendered that has changed since the last data transmission; or, when the current moment is a full transmission moment, all the point cloud data to be rendered is transmitted to the image rendering module. The point cloud data to be rendered, transmitted to the image rendering module, is used for this image rendering.

[0151] In some possible implementations, device 70 further includes: The confirmation module is configured to determine the weight of the point cloud data to be rendered for any object. The weight is positively correlated with the amount of point cloud data to be rendered for the object and inversely correlated with the image volume of the object. A new module has been added, configured to increase the amount of point cloud data for an object when the weight is less than the weight threshold, so that the weight of the point cloud data to be rendered in the image of the object reaches the weight threshold.

[0152] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0153] This disclosure also provides an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to: perform the steps of the image rendering method provided in this disclosure.

[0154] This disclosure also provides a non-transitory computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of the image rendering method provided in this disclosure.

[0155] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the image rendering method provided in this disclosure.

[0156] Figure 8 This is a block diagram illustrating a vehicle 800 according to an exemplary embodiment. For example, vehicle 800 can be a hybrid vehicle, a non-hybrid vehicle, an electric vehicle, a fuel cell vehicle, or other types of vehicle. Vehicle 800 can be an intelligent driving vehicle, a semi-intelligent driving vehicle, or a non-intelligent driving vehicle. For example, vehicle 800 can acquire environmental information about its surroundings through a perception system 820, and based on the analysis of the surrounding environmental information, derive an intelligent driving strategy to achieve fully intelligent driving, or present the analysis results to the user to achieve semi-intelligent driving.

[0157] Reference Figure 8 The vehicle 800 may include various subsystems, such as an infotainment system 810, a perception system 820, a decision control system 830, a drive system 840, and a computing platform 850. The vehicle 800 may also include more or fewer subsystems, and each subsystem may include multiple components. Furthermore, each subsystem and each component of the vehicle 800 can be interconnected via wired or wireless means.

[0158] In some embodiments, the infotainment system 810 may include a communication system, an entertainment system, and a navigation system, etc.

[0159] The perception system 820 may include various sensors for sensing information about the environment surrounding the vehicle 800. For example, the perception system 820 may include a global positioning system, an inertial measurement unit, a lidar, millimeter-wave radar, ultrasonic radar, and a camera device. For instance, lidar uses laser light to sense objects in the environment where the vehicle 800 is located. In some embodiments, lidar may include one or more laser sources, a laser scanner, and one or more detectors, as well as other system components. Millimeter-wave radar uses radio signals to sense objects in the environment surrounding the vehicle 800. In some embodiments, in addition to sensing objects, millimeter-wave radar may also be used to sense the speed and / or direction of travel of objects. Ultrasonic radar may use ultrasonic signals to sense objects around the vehicle 800.

[0160] The decision control system 830 may include a computing system, a vehicle controller, a steering system, a throttle, and a braking system.

[0161] The drive system 840 may include components that provide powered motion to the vehicle 800. In one embodiment, the drive system 840 may include an engine, an energy source, a transmission system, and wheels. The engine may be one or a combination of internal combustion engines, electric motors, and compressed air engines. The engine is capable of converting energy provided by the energy source into mechanical energy.

[0162] Some or all of the functions of the vehicle 800 are controlled by a computing platform 850. The computing platform 850 may include at least one processor 851 and a memory 852, the processor 851 being able to execute instructions 853 stored in the memory 852.

[0163] Processor 851 can be any conventional processor. Processors may also include graphics processing units (GPUs), field programmable gate arrays (FPGAs), systems on chips (SOCs), application-specific integrated circuits (ASICs), or combinations thereof.

[0164] The memory 852 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0165] In addition to instruction set 853, memory 852 can also store data, such as road maps, route information, vehicle position, direction, speed, and other data. The data stored in memory 852 can be used by computing platform 850.

[0166] In this embodiment of the disclosure, processor 851 may execute instruction 853 to complete all or part of the steps of the above-described image rendering method.

[0167] Some embodiments of this disclosure also provide a chip system, such as Figure 9 As shown, the chip system includes at least one processor 901 and at least one interface circuit 902. The processor 901 and the interface circuit 902 are interconnected via lines. For example, the interface circuit 902 can be used to receive signals from other devices (e.g., the memory of an electronic device). As another example, the interface circuit 902 can be used to send signals to other devices (e.g., the processor 901). Exemplarily, the interface circuit 902 can read instructions stored in memory and send those instructions to the processor 901. When the instructions are executed by the processor 901, the image rendering apparatus can perform the various steps in the above-described image rendering method embodiments. Of course, the chip system may also include other discrete devices, and some embodiments of this disclosure do not specifically limit this.

[0168] In some embodiments of this disclosure, the interface circuit 902 can acquire data, program instructions, and / or information from the internal storage area of ​​the chip system; it can also acquire data, program instructions, and / or information from outside the chip system.

[0169] Optionally, the chip system may also include a memory for storing necessary computer programs and data.

[0170] Figure 10 This is a block diagram illustrating an electronic device according to an exemplary embodiment. For example, the electronic device 1000 may be a vehicle, server, mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0171] Reference Figure 10 The electronic device 1000 may include one or more of the following components: processing component 1002, memory 1004, power supply component 1006, multimedia component 1008, audio component 1010, input / output interface 1012, sensor component 1014, and communication component 1016.

[0172] Processing component 1002 typically controls the overall operation of electronic device 1000, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 1002 may include one or more processors 1020 to execute instructions to complete all or part of the steps of the image rendering method described above. Furthermore, processing component 1002 may include one or more modules to facilitate interaction between processing component 1002 and other components. For example, processing component 1002 may include a multimedia module to facilitate interaction between multimedia component 1008 and processing component 1002.

[0173] Memory 1004 is configured to store various types of data to support the operation of electronic device 1000. Examples of such data include instructions for any application or method operating on electronic device 1000, contact data, phonebook data, messages, pictures, videos, etc. Memory 1004 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0174] The power supply assembly 1006 provides power to the various components of the electronic device 1000.

[0175] Multimedia component 1008 includes a screen that provides an output interface between electronic device 1000 and user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 1008 includes a front-facing camera and / or a rear-facing camera. When electronic device 1000 is in an operating mode, such as a shooting mode or video mode, the front-facing camera and / or rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0176] Audio component 1010 is configured to output and / or input audio signals. For example, audio component 1010 includes a microphone (MIC) configured to receive external audio signals when electronic device 1000 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 1004 or transmitted via communication component 1016. In some embodiments, audio component 1010 also includes a speaker for outputting audio signals.

[0177] Input / output interface 1012 provides an interface between processing component 1002 and peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, start buttons, and lock buttons.

[0178] Sensor assembly 1014 includes one or more sensors for providing state assessment of various aspects of electronic device 1000. For example, sensor assembly 1014 can detect the on / off state of electronic device 1000, the relative positioning of components such as the display and keypad of electronic device 1000, changes in position of electronic device 1000 or a component of electronic device 1000, the presence or absence of user contact with electronic device 1000, the orientation or acceleration / deceleration of electronic device 1000, and temperature changes of electronic device 1000. Sensor assembly 1014 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 1014 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications.

[0179] The communication component 1016 is configured to facilitate wired or wireless communication between the electronic device 1000 and other devices.

[0180] In an exemplary embodiment, the electronic device 1000 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the image rendering method described above.

[0181] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 1004 including instructions, which can be executed by a processor 1020 of an electronic device 1000 to complete the image rendering method described above. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0182] Those skilled in the art will also understand that the various illustrative logical blocks and steps listed in the embodiments of this application can be implemented by electronic hardware, computer software, or a combination of both. Whether such functionality is implemented through hardware or software depends on the specific application and the overall system design requirements. Those skilled in the art can implement the described functionality using various methods for each specific application, but such implementation should not be construed as exceeding the scope of protection of the embodiments of this application.

[0183] Furthermore, the term “exemplary” is used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as “exemplary” is not necessarily to be construed as advantageous compared to other aspects or designs. Rather, the use of the term “exemplary” is intended to present the concept in a concrete manner. As used herein, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless otherwise specified or clear from the context, “X applies A or B” is intended to mean any of the natural inclusive arrangements. That is, “X applies A or B” satisfies any of the foregoing instances if X applies A; X applies B; or both X applies A and B. Additionally, unless otherwise specified or clear from the context to refer to the singular form, the articles “a” and “an” as used in this application and the appended claims are generally understood to mean “one or more.”

[0184] Similarly, although this disclosure has been shown and described with respect to one or more implementations, equivalent variations and modifications will occur to those skilled in the art upon reading and understanding this specification and the accompanying drawings. This disclosure includes all such modifications and variations and is limited only by the scope of the claims. In particular, with respect to the various functions performed by the components described above (e.g., elements, resources, etc.), unless otherwise indicated, the terminology used to describe such components is intended to correspond to any component (functionally equivalent) that performs the specific function of the described component, even if structurally not equivalent to the disclosed structure. Furthermore, although specific features of this disclosure may have been disclosed with respect to only one of several implementations, such features may be combined with one or more other features of other implementations, as may be desired and advantageous to any given or particular application. Moreover, with regard to the terms “comprising,” “owning,” “having,” “having,” or variations thereof as used in the detailed description or claims, such terms are intended to be inclusive in a manner similar to the term “including.”

[0185] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

[0186] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An image rendering method, characterized in that, The method includes: Acquire the first point of cloud data detected by the mobile device at the current moment; Acquire second point cloud data detected by the mobile device at a historical moment; wherein, the historical moment refers to a moment prior to the current moment; The second point cloud data is mapped to the location coordinate system of the mobile device at the current moment to determine the historical point cloud data; The first point cloud data is fused with the historical point cloud data to obtain the point cloud data to be rendered; Image rendering is performed based on the point cloud data to be rendered.

2. The method according to claim 1, characterized in that, The second point cloud data includes at least backward data, which is point cloud data of objects detected at the historical moment and located behind the direction of movement of the mobile device at the current moment.

3. The method according to claim 1, characterized in that, The spatial range described by the position coordinate system includes multiple regions, and different regions correspond to different point cloud fusion methods. The step of fusing the first point cloud data with the historical point cloud data to obtain the point cloud data to be rendered includes: For any of the aforementioned regions, determine the first data of the first point cloud data that is located within the region and the second data of the historical point cloud data that is located within the region; The first data and the second data are fused according to the fusion method corresponding to the region to obtain fused data; The fused data from each region is combined to obtain the point cloud data to be rendered.

4. The method according to claim 3, characterized in that, The step of fusing the first data and the second data according to the fusion method corresponding to the region to obtain fused data includes: Determine the growth rate of the amount of first data within the first point cloud data area compared to the amount of second data within the historical point cloud data area; If the growth rate is greater than the growth rate threshold corresponding to the region, the second data in the region is replaced with the first data, and the first data is used as the fused data of the region.

5. The method according to claim 4, characterized in that, The spatial range described by the position coordinate system is a three-dimensional spatial range. The growth rate of the amount of first data in the first point cloud data within the region compared to the amount of second data in the historical point cloud data within the region includes: The three-dimensional spatial range is mapped onto a two-dimensional space on a horizontal plane, and the two-dimensional space is divided into multiple grids; For any given grid, determine the growth rate of the first data volume of the first point cloud data within the grid compared to the growth rate of the second data volume of the historical point cloud data within the grid.

6. The method according to claim 4, characterized in that, The magnitude of the growth rate threshold corresponding to each region is inversely correlated with the distance between the region and the mobile device.

7. The method according to claim 1, characterized in that, The step of fusing the first point cloud data with the historical point cloud data to obtain the point cloud data to be rendered includes: Determine the updated region and the fixed region within the spatial range described by the position coordinate system; The first data of the first point cloud data located within the updated area and the second data of the historical point cloud data located within the updated area are merged to obtain merged data; The third data, which is located within the fixed area of ​​the fused data and the historical point cloud data, is used as the point cloud data to be rendered.

8. The method according to claim 3, characterized in that, The step of fusing the first data and the second data according to the fusion method corresponding to the region to obtain fused data includes: Determine the amount of first data in the region of the first point cloud data and the amount of second data in the region of the historical point cloud data; If the data volume of the first data and the data volume of the second data are both less than the quantity threshold, the first data and the second data are superimposed, and the superimposed data is used as the fused data of the region.

9. The method according to any one of claims 1-8, characterized in that, The step of rendering the image based on the point cloud data to be rendered includes: The point cloud data to be rendered is transmitted to the image rendering module of the mobile device; wherein, when the current time is not a full transmission time, incremental point cloud data is transmitted to the image rendering module, and the incremental point cloud data is the data in the point cloud data to be rendered that has changed since the last data transmission; or, when the current time is a full transmission time, all the point cloud data to be rendered is transmitted to the image rendering module. The point cloud data to be rendered, transmitted to the image rendering module, is used for this image rendering.

10. The method according to any one of claims 1-8, characterized in that, Before rendering the image based on the point cloud data to be rendered, the method further includes: Determine the weight of the point cloud data to be rendered corresponding to any object. The weight is positively correlated with the amount of point cloud data to be rendered corresponding to the object and inversely correlated with the image volume of the object. If the weight is less than the weight threshold, the amount of point cloud data of the object is increased so that the weight of the point cloud data to be rendered for rendering the image of the object reaches the weight threshold.

11. An image rendering apparatus, characterized in that, The device includes: The first acquisition module is configured to acquire the first point cloud data detected by the mobile device at the current moment; The second acquisition module is configured to acquire second point cloud data detected by the mobile device at a historical time; wherein, the historical time refers to a time before the current time; The determination module is configured to map the second point cloud data to the location coordinate system of the mobile device at the current time to determine the historical point cloud data; The fusion module is configured to fuse the first point cloud data with the historical point cloud data to obtain point cloud data to be rendered. The rendering module is configured to perform image rendering based on the point cloud data to be rendered.

12. The apparatus according to claim 11, characterized in that, The second point cloud data includes at least backward data, which is point cloud data of objects detected at the historical moment and located behind the direction of movement of the mobile device at the current moment.

13. The apparatus according to claim 11, characterized in that, The spatial range described by the position coordinate system includes multiple regions, and different regions correspond to different point cloud fusion methods. The fusion module is specifically configured as follows: For any of the aforementioned regions, determine the first data of the first point cloud data that is located within the region and the second data of the historical point cloud data that is located within the region; The first data and the second data are fused according to the fusion method corresponding to the region to obtain fused data; The fused data from each region is combined to obtain the point cloud data to be rendered.

14. The apparatus according to claim 11, characterized in that, The fusion module is specifically configured as follows: Determine the updated region and the fixed region within the spatial range described by the position coordinate system; The first data of the first point cloud data located within the updated area and the second data of the historical point cloud data located within the updated area are merged to obtain merged data; The third data, which is located within the fixed area of ​​the fused data and the historical point cloud data, is used as the point cloud data to be rendered.

15. A vehicle, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured as follows: The steps of performing the method according to any one of claims 1-10.

16. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured as follows: The steps of performing the method according to any one of claims 1-10.

17. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program performs the steps of the method described in any one of claims 1-10.

18. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1-10.

19. A chip, characterized in that, It includes a processor and an interface, the processor being configured to read instructions to perform the steps of the method according to any one of claims 1-10.

Images