Space intelligence-oriented end-cloud collaborative instant positioning and mapping method and device

Abstract

This disclosure provides a method and apparatus for edge-cloud collaborative real-time localization and mapping for spatial intelligence, relating to the field of computer technology, particularly artificial intelligence, image processing, visual localization and mapping, and edge-cloud collaborative computing. The scheme includes: acquiring an image to be processed from an image acquisition device; acquiring motion vectors of multiple coding units of the image to be processed; determining the cumulative motion of the image to be processed relative to a target transmitted frame based on the motion vectors of the multiple coding units; comparing the cumulative motion with a preset motion range; filtering out the image to be processed if the cumulative motion is lower than the minimum value of the preset motion range; encoding the image to be processed if the cumulative motion is within the preset motion range to obtain a first encoded frame of the image to be processed; and sending the first encoded frame of the image to be processed to the cloud, so that the cloud processes the image to be processed using a feedforward real-time localization and mapping neural network model.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to the fields of artificial intelligence, image processing, visual positioning and mapping, and edge-cloud collaborative computing. Background Technology

[0002] Simultaneous Localization and Mapping (SLAM) is a core technology for spatial intelligence applications such as Augmented Reality (AR), Virtual Reality (VR), robot navigation, autonomous driving, and digital twins. With the rapid development of spatial computing and embodied intelligence technologies, SLAM has become the perceptual infrastructure supporting intelligent agents and their environment in spatial understanding, and its output quality directly determines the security and interactive intelligence level of upper-layer applications.

[0003] Classic edge-side SLAM solutions require the entire geometric computation process, including feature extraction, bundle adjustment, and loop closure detection, to be completed on mobile devices, resulting in high power consumption and sparse output. Summary of the Invention

[0004] This disclosure provides a method and apparatus for real-time positioning and mapping via edge-cloud collaboration for spatial intelligence, in order to solve or alleviate one or more technical problems in the prior art.

[0005] Firstly, this disclosure provides a method for real-time positioning and mapping via edge-cloud collaboration for spatial intelligence, including: Acquire the image to be processed from the image acquisition device; Obtain the motion vectors of multiple coding units of the image to be processed; Based on the motion vectors of the multiple coding units, the cumulative motion of the image to be processed relative to the target transmitted frame is determined; the target transmitted frame is the frame image that is closest to the image to be processed in time in the frame sequence transmitted to the cloud. Compare the cumulative exercise volume with a preset exercise volume range; If the cumulative motion amount is lower than the lowest value of the preset motion amount range, the image to be processed is filtered out. When the cumulative motion amount is within the preset motion amount range, the image to be processed is encoded to obtain the first encoded frame of the image to be processed. The first encoded frame of the image to be processed is sent to the cloud so that the cloud processes the image to be processed through a feedforward real-time localization and mapping neural network model.

[0006] Secondly, this disclosure provides an edge-cloud collaborative real-time positioning and mapping device for spatial intelligence, comprising: The acquisition module is used to acquire the image to be processed acquired by the image acquisition device; and to acquire the motion vectors of multiple coding units of the image to be processed. The motion determination module is used to determine the cumulative motion of the image to be processed relative to the target transmitted frame based on the motion vectors of the plurality of coding units; the target transmitted frame is the frame image that is closest in time to the image to be processed in the frame sequence transmitted to the cloud; The comparison module is used to compare the cumulative exercise volume with a preset exercise volume range; A filtering module is used to filter out the image to be processed when the cumulative motion amount is lower than the minimum value of the preset motion amount range; The encoding module is used to encode the image to be processed when the cumulative motion amount is within the preset motion amount range, so as to obtain the first encoded frame of the image to be processed. The sending module is used to send the first encoded frame of the image to be processed to the cloud, so that the cloud can process the image to be processed through a feedforward real-time localization and mapping neural network model.

[0007] Thirdly, an electronic device is provided, comprising: At least one processor; and The memory is communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform any of the methods described in the present disclosure.

[0008] Fourthly, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform any of the methods according to embodiments of the present disclosure.

[0009] Fifthly, a computer program product is provided, including a computer program that, when executed by a processor, implements any of the methods according to embodiments of the present disclosure.

[0010] The beneficial effects of the technical solution provided in this disclosure include at least the following: It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments provided according to this disclosure and should not be construed as limiting the scope of this disclosure.

[0012] Figure 1 This is a flowchart illustrating an edge-cloud collaborative SLAM method for spatial intelligence according to an embodiment of the present disclosure. Figure 2 This is a flowchart illustrating the time alignment and extrapolation of inference results according to an embodiment of the present disclosure; Figure 3 This is a schematic diagram illustrating the implementation process of an edge-cloud collaborative SLAM method for spatial intelligence according to an embodiment of the present disclosure; Figure 4 This is a schematic diagram of an edge-cloud collaborative SLAM system for spatial intelligence according to an embodiment of the present disclosure; Figure 5 This is a schematic diagram of the structure of an edge-cloud collaborative SLAM device for spatial intelligence according to an embodiment of the present disclosure; Figure 6 This is a block diagram of an electronic device used to implement the edge-cloud collaborative SLAM method for spatial intelligence according to embodiments of this disclosure. Detailed Implementation

[0013] The present disclosure will now be described in further detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0014] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0015] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this disclosure, "multiple" means two or more, unless otherwise explicitly specified.

[0016] It should be noted that, unless it is explicitly stated that there is a sequential order of execution between different operations, or that there is a sequential order of execution between different operations in terms of technical implementation, the execution order between multiple operations may not be significant, and multiple operations may be executed simultaneously.

[0017] In related technologies, edge-cloud collaborative SLAM implementation schemes require the edge device to complete the entire geometric computation process, including feature extraction, bundle adjustment, and loop closure detection. This scheme still requires the edge device to handle the entire tracking thread, resulting in high engineering complexity. Currently, there is no solution that simultaneously satisfies both lightweight edge-cloud system implementation of SLAM technology and high-density output.

[0018] In view of this, the present disclosure provides an edge-cloud collaborative SLAM method for spatial intelligence, which can achieve a lightweight edge-cloud collaborative solution with dense output without performing any SLAM geometric calculations on the edge side.

[0019] like Figure 1 The diagram shown is a flowchart of an edge-cloud collaborative SLAM method for spatial intelligence according to an embodiment of this disclosure, including the following: S101, acquire the image to be processed acquired by the image acquisition device.

[0020] An image acquisition device refers to a sensor device mounted on an edge device for capturing visual information about the environment. In this embodiment, the edge device is lightweight and can support any type of image acquisition device. For example, the image acquisition device can be a monocular RGB camera, that is, it only uses a common RGB lens for image acquisition, without relying on additional hardware devices such as depth cameras, time-of-flight (ToF) sensors, structured light sensors, binocular stereo cameras, or LiDAR. Monocular RGB cameras have advantages such as low cost, high availability, and low power consumption, and can be widely used in various edge devices such as smartphones, AR / VR headsets, robots, and drones.

[0021] The image to be processed refers to the image frame acquired in real time by the image acquisition device. Any frame image acquired by the image acquisition device can be used as the image to be processed. Therefore, the image to be processed is the original image data in the image sequence acquired by the image acquisition device.

[0022] During implementation, the edge device continuously acquires image sequences at the target resolution and frame rate using an image acquisition unit to obtain the image to be processed. It should be noted that the acquisition resolution and frame rate are configurable parameters and can be dynamically adjusted based on the computing load and network conditions reported by the cloud side. For example, when network bandwidth is sufficient and cloud computing power is idle, a higher resolution and frame rate can be used for acquisition to obtain richer image details for cloud-side SLAM inference; when network bandwidth is limited or the cloud side load is high, the resolution and frame rate can be dynamically reduced to ensure the real-time performance and stability of uplink transmission and avoid SLAM inference lag caused by data transmission delays.

[0023] S102, obtain the motion vectors of multiple coding units of the image to be processed.

[0024] In this context, a coding unit (CU) refers to the basic unit used for prediction, transform, quantization, and entropy coding under video coding standards (such as H.264 / H.265 / H.266), including but not limited to macroblocks (MB) in H.264 / Advanced Video Coding (AVC), coding blocks (CB) or coding units (CU) in H.265 / High Efficiency Video Coding (HEVC) and Versatile Video Coding (VVC), and superblocks and their subdivisions in AV1.

[0025] A motion vector (MV) refers to the spatial displacement of a coding unit relative to its best-matching block in a reference frame during inter-frame prediction in video coding. A motion vector is typically represented as a two-dimensional vector (dx, dy), where dx represents the horizontal displacement (in pixels) and dy represents the vertical displacement (in pixels).

[0026] During implementation, the video encoder calculates motion vectors for each coding unit when performing inter-frame prediction during the encoding process. For example, for the H.265 / HEVC or VVC coding standard, the motion vectors of each coding unit can be obtained from the Coding Tree Unit (CTU) and its subdivided coding units and prediction units (PU); for the H.264 / AVC coding standard, the motion vectors of each block can be obtained from the macroblock and its sub-blocks; for the AV1 standard, the motion vectors of each block can be obtained from the superblock and its subdivided coding blocks.

[0027] In practice, motion vectors can be obtained in the following ways: directly reading the motion vector data calculated internally from the encoder's motion estimation module. In this case, the obtained motion vector is the original motion vector without entropy encoding and bitstream packaging, which has the highest numerical accuracy.

[0028] It is understood that the motion vector is obtained synchronously by the terminal device during the encoding of the image to be processed. This disclosure directly reuses the motion vector without performing optical flow estimation or feature matching on the image separately, thus not introducing additional geometric computation overhead, in order to meet the requirements of lightweight terminal.

[0029] S103, based on the motion vectors of multiple coding units, determine the cumulative motion of the image to be processed relative to the target transmitted frame; the target transmitted frame is the frame image that is closest in time to the image to be processed in the frame sequence transmitted to the cloud.

[0030] In other words, the target transmitted frame is the image frame that was last actually encoded and successfully uploaded to the cloud.

[0031] During implementation, the motion vectors of all valid coded units of the image to be processed are statistically aggregated to characterize the inter-frame motion amplitude of the current frame (i.e., the image to be processed) relative to the target transmitted frames.

[0032] In practice, effective coding units can be identified from multiple coding units based on their respective motion vectors, and then the cumulative motion of the image to be processed can be determined based on the effective coding units.

[0033] Specifically, the determination of the effective coding unit may include the following steps: Step A1: Based on the statistics of motion vector magnitude, calculate the motion vector magnitude (magnitude) of each coding unit in the current frame, and perform statistics on the motion vector magnitude of all effective coding units to obtain at least one of the median, mean, or quantile, and obtain the outlier analysis value.

[0034] Step A2: Based on outlier analysis values, identify valid coding units from multiple coding units.

[0035] The following sections will provide detailed explanations of how to remove outliers from multiple coding units based on the median, mean, and quantile.

[0036] 1) Outlier removal based on median When the median is used as the statistical value, the median M of the motion vector amplitude of all coding units is calculated. The motion vector amplitude V of each coding unit is compared with the median M to obtain a value representing the degree of deviation. When this value exceeds a preset deviation threshold, the corresponding coding unit is identified as an outlier and removed from the multiple coding units of the image to be processed.

[0037] In implementation, the absolute deviation can be obtained by calculating the absolute value of the difference between the motion vector amplitude and the median M of each effective coding unit; the median of all absolute deviations is taken and denoted as MAD; the threshold is set to be equal to the product of the coefficient α and MAD, where α is a preset positive constant. For a coding unit with a motion vector amplitude of V, if |V - M|>α·MAD, then the coding unit is identified as an outlier.

[0038] In another example, the deviation threshold can also be set directly as a fixed proportion of the median M. If |V - M|>β·M, then the corresponding coding unit is determined as an outlier, where β is a preset proportion coefficient.

[0039] 2) Identifying outliers based on the mean When the mean is used as the statistic, the arithmetic mean μ of the motion vector amplitudes of all coding units is calculated, and the standard deviation σ can be further calculated. Outlier criteria are constructed using the mean and standard deviation: for a coding unit with a motion vector amplitude of V, if its standardized deviation |V - μ| is greater than a preset multiple threshold k·σ, then the coding unit is identified as an outlier, where k is a preset positive constant, typically 3.

[0040] Alternatively, the ratio of the motion vector amplitude to the mean can be used as a criterion. If V / μ exceeds the upper limit of the preset ratio or is lower than the lower limit of the preset ratio, the corresponding coding unit is identified as an outlier.

[0041] 3) Identifying outliers based on quantiles When quantiles are used as statistical measures, at least one quantile is used to define the range of normal motion vector amplitudes, and coding units whose motion vector amplitudes fall outside the range are identified as outliers.

[0042] In one example, the upper quartile Q3 and lower quartile Q1 of the motion vector magnitudes of multiple coding units in the image to be processed are taken, and the interquartile range IQR = Q3 - Q1 is calculated. The normal range is set as [Q1 - γ·IQR, Q3 + γ·IQR], where γ is a preset coefficient, typically 1.5. If the motion vector magnitude V of a certain coding unit is greater than Q3 + γ·IQR, or less than Q1 - γ·IQR, it is identified as an outlier.

[0043] In another example, a single high quantile is used for judgment, such as the 90th percentile P90. Encoding units with motion vector amplitude V greater than δ·P90 are identified as outliers, where δ is a scaling factor greater than 1, which can be set empirically, for example, δ is 1.5 or 2.0. Similarly, low quantiles can be used to detect outliers with abnormally small motion amplitudes.

[0044] In practice, the median, mean, and quantiles can be used individually or in combination as outlier analysis values ​​to identify outliers. When at least two of the median, mean, and quantiles are used as outlier analysis values, outliers can be determined separately based on each parameter. Then, all outliers determined by all parameters are removed from multiple coding units of the image to be processed, and the remaining coding units are considered as valid coding units.

[0045] By proposing solutions for outliers under different statistical measures, we can flexibly adapt to the motion characteristics of different video scenarios, robustly identify coding units with abnormal motion behavior, and provide a reliable basis for subsequent coding optimization.

[0046] Step A3: Based on the effective coding units, determine the cumulative motion of the image to be processed relative to the target transmitted frames.

[0047] For the valid coding units obtained after removing outliers, the cumulative motion of the image to be processed relative to the target transmitted frame is further determined. The motion vector of each coding unit represents the displacement of that coding unit from the target transmitted frame to the image to be processed. The process of determining the cumulative motion is as follows: Obtain the motion vector corresponding to the target transmitted frame for each valid coding unit, calculate the magnitude of the motion vector, and record it as the motion vector magnitude of the valid coding unit. Accumulate the motion vector magnitudes of all valid coding units, and use the accumulated result as the cumulative motion amount.

[0048] Alternatively, the arithmetic mean of the motion vector magnitudes of all valid coding units can be calculated, and the product of this arithmetic mean and the total number of valid coding units can be used as the cumulative motion.

[0049] In scenarios that consider differences in coding unit size, the magnitude of the corresponding motion vector can be weighted and accumulated based on the area of ​​each effective coding unit to obtain the area-weighted cumulative motion.

[0050] The above method uses the effective coding units retained after removing outliers to calculate the cumulative motion, eliminating the interference of coding units with abnormal motion vector amplitudes on frame-level motion intensity estimation. This allows the obtained cumulative motion to more accurately reflect the true overall motion of the image to be processed relative to the target transmitted frames, providing a reliable basis for subsequent coding parameter decisions (such as frame type selection, quantization parameter adjustment, etc.).

[0051] In practice, when determining outliers and cumulative motion, the motion vector can be a forward motion vector, or a backward motion vector can be selected according to actual needs.

[0052] S104 compares the cumulative exercise volume with the preset exercise volume range.

[0053] During implementation, the preset exercise range includes the minimum value (also known as the lower threshold, denoted as T). low ) and the highest value (also known as the upper threshold, denoted as T) high This preset exercise range can be set and adjusted according to the actual application scenario and the needs of the cloud-based SLAM algorithm. For example, when cloud network bandwidth is limited, the range can be increased by T. low This raises the filtering threshold, discarding more redundant frames with low motion, and further saving bandwidth; when the computing power of the cloud-side Graphics Processing Unit (GPU) is idle, the T can be reduced. low This lowers the filtering threshold, allows more frames to be uploaded, and improves perception accuracy.

[0054] S105: If the cumulative motion volume is lower than the minimum value of the preset motion volume range, filter out the image to be processed.

[0055] When the cumulative motion is below the minimum value of the preset motion range, it indicates that the motion of the image to be processed relative to the target transmitted frame is very small. The content of the image to be processed and the target transmitted frame are highly similar, lacking effective parallax, and there is no effective gain for the geometric solution of cloud-side SLAM. Uploading it would waste uplink bandwidth and cloud computing power. Therefore, the image to be processed is directly filtered out on the edge, without encoding or transmission, to save bandwidth and cloud computing resources.

[0056] After filtering out the images to be processed, the next frame of the image to be processed acquired by the image acquisition device is taken as the new image to be processed. The process returns to step S102 to continue to acquire the motion vector of the next frame and calculate and determine the cumulative motion.

[0057] During the filtering process, although the image data of that frame is discarded, the motion vectors generated by the encoder during inter-frame prediction are still used for the continuous accumulation of motion. In other words, the end-to-cloud collaborative system provided in this embodiment of the present disclosure will continuously accumulate the cumulative motion from the target transmitted frame to the current frame until the cumulative motion reaches T. low Then the next frame is transmitted, thus ensuring sufficient parallax between adjacent transmitted frames while avoiding the transmission of nearly identical images.

[0058] S106, when the cumulative motion amount is within the preset motion amount range, the image to be processed is encoded to obtain the first encoded frame of the image to be processed.

[0059] When the cumulative motion amount is within the preset motion amount range, it indicates that the image to be processed has reasonable motion changes relative to the target transmitted frame and has moderate and clear parallax. This image to be processed is suitable as an input frame for SLAM, providing effective geometric constraints for cloud-side SLAM without causing cross-frame matching failure due to excessive motion. Such image frames are defined as suitable frames and are encoded normally according to the original strategy.

[0060] During implementation, the terminal device calls the hardware video encoder to encode the image to be processed into a bitstream frame of a standard video encoding protocol (such as H.264 / H.265 / AV1, etc.), thus obtaining the first encoded frame.

[0061] S107, the first encoded frame of the image to be processed is sent to the cloud so that the cloud can process the image to be processed through a feedforward SLAM neural network model.

[0062] Feedforward SLAM neural network models are models with specific structural features, including: (a) A shared visual feature encoder for extracting multi-level visual feature representations from each frame of input image; (b) Cross-frame aggregation backbone, used to establish spatial-temporal correspondences between multiple input images within the same sliding window based on the self-attention mechanism; (c) Multi-task prediction head, including: camera pose prediction head, used to output the camera pose of each frame of input image; dense depth prediction head, used to output the depth map of each frame of input image; 3D point cloud prediction head, used to output dense 3D point cloud in world coordinate system.

[0063] In implementation, the feedforward SLAM neural network model can be any of the following models or an improved version thereof, including: Visual Geometry Grounded Transformer (VGGT), Dense and Unconstrained Stereo 3D Reconstruction (DUSt3R), Multi-View Stereo 3D Reconstruction (MASt3R), Continuous Updating Transformer for 3D Reconstruction (CUT3R), and other end-to-end multi-view geometric neural network models with similar input-output formats.

[0064] In implementation, the terminal device sends the first encoded frame of the image to be processed to the cloud via the uplink channel. The cloud uses a video decoding module to decode the first encoded frame into an image frame in RGB or YUV format in real time, and then sends the decoded image frame into a feedforward SLAM neural network model for processing. The uplink channel uses encapsulation based on Web Real-Time Communication (WebRTC), Real-Time Streaming Protocol (RTSP), Secure Reliable Transport (SRT), or a low-latency transport protocol based on User Datagram Protocol (UDP). The content transmitted through this uplink channel is the pixel information of the image frame compressed using the aforementioned standard video encoding. The video decoding module includes, but is not limited to, an NVIDIA NVDEC hardware decoder or an FFmpeg software decoder.

[0065] This feedforward SLAM neural network model can perform single forward inference in a sliding window manner. The cloud transmits the inference results of each frame (including at least one of pose, depth map, and 3D point cloud, as well as their respective timestamps) back to the edge via the downlink channel for use by the local rendering and application modules on the edge.

[0066] In this embodiment, by utilizing motion vectors generated during video encoding at the terminal side, lightweight perception of motion in image frames is achieved, and the decision on whether to transmit the image frame is adaptively based on this motion. Adaptively selecting which image frames to encode and transmit effectively reduces redundant data transmission in edge-cloud collaborative SLAM, lowers network bandwidth usage and cloud computing load, while ensuring the transmission of key frames, thereby optimizing the overall efficiency and stability of edge-cloud collaborative SLAM. In summary, in this embodiment, the terminal side only uses accumulated motion to filter frames, achieving lightweight terminal-side processing, while the cloud outputs inference results through the SLAM neural network model, enabling dense output. Therefore, this embodiment provides an edge-cloud collaborative SLAM solution that achieves both lightweight terminal-side processing and dense output.

[0067] When the accumulated motion is too large, the difference between the image to be processed and the target frame that has already been transmitted is significant. Directly transmitting the image to be processed may make it difficult for cloud-based SLAM to establish reliable inter-frame matching. In this case, several interpolated frames are generated through interpolation to cope with scenarios where the inter-frame motion is too large due to rapid camera movement. Therefore, this method also includes the following steps: Step B1: If the cumulative motion amount is higher than the highest value of the preset motion amount range, perform an interpolation operation based on the image to be processed and the target transmitted frame to obtain at least one interpolated frame.

[0068] Specifically, step B1 includes steps B11-B14: Step B11: Determine the number of interpolation frames N based on the ratio of the cumulative motion amount to the highest value of the preset motion amount interval; N is an integer greater than or equal to 1, and N is proportional to the ratio.

[0069] During implementation, the greater the ratio of the cumulative motion amount to the highest value of the preset motion amount range, the more interpolation frames need to be inserted in order to reduce the motion interval between adjacent transmission frames to a suitable range.

[0070] Step B12: For the k-th interpolated frame in the N interpolated frames, determine the interpolation coefficients of the k-th interpolated frame; the k-th interpolated frame is any frame in the N interpolated frames; wherein the interpolation coefficients are proportional to k and inversely proportional to N.

[0071] During implementation, for the k-th interpolated frame (k=1,2,...,N) among the N interpolated frames, the interpolation coefficient α of that frame is determined. k Specifically, the interpolation coefficient α can be set. k =k / (N+1), thus ensuring that each interpolated frame is evenly distributed between the target transmitted frame and the image to be processed.

[0072] Step B13: Linearly scale the motion vectors of multiple coding units based on the interpolation coefficients to obtain the intermediate motion vectors of each coding unit in the image to be processed relative to the target transmitted frame.

[0073] In practice, the motion vectors of multiple coding units (which can be valid coding units) in the current image to be processed relative to the target transmitted frames can be denoted as MV. total The intermediate motion vector of each coding unit (which can be a valid coding unit) in the image to be processed relative to the target transmitted frame can be denoted as MV. intermediate,k For the k-th interpolated frame, which is located at the time position between the target transmitted frame and the current image to be processed, its corresponding displacement should be a part of the total displacement, and the scaling factor is the interpolation factor α. k Therefore, the intermediate motion vector of each coded unit in the image to be processed relative to the target transmitted frame can be denoted as MV. intermediate,k =α k ×MV total .

[0074] For example, for each coding unit, the horizontal and vertical components of its motion vector are multiplied by interpolation coefficients αk to obtain the intermediate motion vector of that coding unit in the k-th interpolated frame: MV intermediate,k,x =α k ×MV total,x MV intermediate,k,y =α k ×MV total,y When α k When the value is within the (0,1) interval, the displacement corresponding to the intermediate motion vector is between zero displacement (i.e., the position of the target frame that has been transmitted) and total displacement (i.e., the position of the image to be processed), thus ensuring that the pixel positions of the interpolated frame are evenly distributed in space between the two ends. By using linear scaling, it is possible to ensure that the generated multi-frame interpolation transitions smoothly on the motion trajectory, avoiding jumps or discontinuous motion, and also to utilize the effective computing resources on the edge to reduce the complexity of interpolation operations.

[0075] Step B14: Based on the intermediate motion vector, perform linear motion compensation displacement on the pixels of the corresponding coding unit in the target transmitted frame to generate the k-th interpolated frame.

[0076] During implementation, the intermediate motion vector (MV) calculated in step B13 is used. intermediate,k Using the target transmitted frame as a reference image, linear motion compensation displacement is performed on the pixels of each coded unit in that frame. Specifically, for any pixel position (x, y) in the target transmitted frame, it is moved to the latest pixel position (x', y'), where x' = x + MV. intermediate,k,x y'=y+MV intermediate,k,yThis generates the latest pixel position corresponding to the interpolated frame. By traversing all coded units in the target's transmitted frames and performing the linear motion compensation displacement operation described above, the complete k-th interpolated frame can be generated.

[0077] In this embodiment, by performing linear interpolation based on motion vectors and adaptively generating one or more interpolated frames, the motion interval between adjacent transmission frames is reduced and falls back to the appropriate input range for the cloud-side SLAM neural network. This method is simple and efficient. On the one hand, it is compatible with motion vectors in the video encoding process, requiring no additional complex optical flow calculations, making it suitable for real-time execution on terminal devices. On the other hand, it provides a denser temporal sampling input for the cloud-side SLAM neural network, which helps improve the smoothness and robustness of pose estimation.

[0078] Step B2: After encoding the image to be processed, a first encoded frame of the image to be processed is obtained; and after encoding at least one interpolation frame, a second encoded frame of at least one interpolation frame is obtained.

[0079] In implementation, the terminal device invokes the on-side hardware video encoder to encode the image to be processed and one or more interpolated frames. Specifically, the image to be processed is encoded into a first encoded frame, and each interpolated frame is encoded as a second encoded frame. The encoding uses one or more of the aforementioned standard video encoding protocols, such as H.264, H.265, and AV1, for real-time encoding.

[0080] To meet the processing requirements of cloud-based SLAM, the encoding adopts a low-latency mode, including but not limited to the following methods: disabling or minimizing B-frames, shortening the Group of Pictures (GOP) length, enabling Constant Bit Rate (CBR) or Low-Latency Variable Bit Rate (VBR), to ensure encoding efficiency and real-time transmission.

[0081] It should be noted that the number N of interpolated frames is determined in step B11 based on the highest value T between the cumulative motion and the preset motion interval. high The ratio is dynamically determined, therefore, the number of second-coded frames is the same as N. When N=1, one interpolated frame and its corresponding second-coded frame are generated; when N>1, multiple interpolated frames and their respective corresponding second-coded frames are generated. It should be noted that the multiple interpolated frames are encoded sequentially in chronological order and arranged in the bitstream according to their corresponding timestamps to ensure that the image sequence can be reconstructed in the correct temporal order after decoding in the cloud.

[0082] Step B3: Send the first and second encoded frames to the cloud.

[0083] In practice, the terminal device sends the first and second encoded frames together to the cloud via the uplink channel. The uplink channel uses the aforementioned transmission protocol for encapsulation and transmission. The transmission order of the first and second encoded frames can be selected based on the actual application scenario and the cloud decoding strategy.

[0084] In this embodiment of the disclosure, by generating and transmitting interpolated frames when the amount of motion is too large, smooth motion transition information is provided to the cloud, which helps the cloud SLAM to achieve more stable and accurate pose estimation and map building in the process of feedforward neural network processing, and avoids tracking failure caused by excessive motion jumps between frames.

[0085] After filtering out the images to be processed, as described above, the process also includes: taking the next frame of the image to be processed acquired by the image acquisition device as the new image to be processed, and returning to execute the step of obtaining the motion vectors of multiple coding units of the image to be processed.

[0086] Understandably, after filtering out the images to be processed, the terminal device continues to acquire the next frame as the new image to be processed, and returns to step S102 to continuously filter image frames. In this way, through the cyclic feedback mechanism, the terminal can continuously acquire the cumulative motion between frames while continuously acquiring images, thus realizing dynamic closed-loop control.

[0087] To further improve coding efficiency and network bandwidth utilization, this embodiment of the disclosure also includes a preprocessing operation based on the motion vector of the coding unit. Specifically, it can be implemented as follows: Step C1: Based on the motion vectors of multiple coding units, outliers in multiple coding units are removed to obtain optimized coding units.

[0088] In implementation, the method for determining outliers is the same as described above, and will not be repeated here. The optimized coding unit obtained here can be understood as the effective coding unit described above.

[0089] Step C2: Based on the optimized motion vectors of the coding units, determine the camera motion type for the image to be processed.

[0090] During implementation, based on the motion vectors of the optimized coding units, the global motion patterns of the motion vectors are analyzed and classified to identify the camera motion type of the current frame relative to the reference frame.

[0091] Camera motion type is used to guide subsequent coding optimization strategies. Different camera motion types correspond to different geometric transformation models. Choosing an appropriate geometric transformation model can enable the encoder to describe the motion vector of the entire frame or region with fewer parameters, thereby significantly reducing coding bit overhead and saving network transmission resources.

[0092] In practice, camera motion types may include at least one of the following: primarily translation, primarily rotation, primarily zoom, a combination of rotation and zoom, pitch or panning, pure rotation, etc. This disclosure is capable of identifying geometrically degenerate motions such as pure rotation.

[0093] During implementation, based on the optimized motion vectors, one or more of the following global motion features are calculated, including but not limited to: divergence, curl, orientation consistency, and translation component. After calculating the above global motion features, the camera motion type is identified based on the saliency combination of each global motion feature.

[0094] For example, when the translation component is significant, the divergence and curl are both small, and the directional consistency is strong, the camera motion is determined to be mainly translation; when the curl is significant, the divergence is extremely small, the translation component is approximately zero, and the directional consistency is extremely low (the direction angles are tangentially distributed), the camera motion is determined to be mainly rotation; when the divergence is significant, the curl is extremely small, the translation component is approximately zero, and the directional consistency is relatively low, the camera motion is determined to be mainly scaling; when both curl and divergence are significant, the camera motion is determined to be a combination of rotation and scaling; when the motion vector field exhibits perspective trapezoidal distortion characteristics (i.e., the amplitude of the motion vector in different areas of the image differs significantly and conforms to the perspective projection law), the camera motion is determined to be pitch or pan; when the curl is significant but the translation component and divergence are both approximately zero, the camera motion is determined to be purely rotational.

[0095] In practice, the significance thresholds for each of the above features can be configured according to the actual application scenario. For example, in handheld device shooting scenarios, since frequent hand shaking introduces small rotational movements, the threshold for rotation determination can be appropriately increased to avoid misjudging the tiny rotations superimposed on normal translational movements as the main type of rotation; while in drone aerial photography scenarios, due to the high flight stability, the threshold can be appropriately decreased to more sensitively capture rotational movements.

[0096] Encoding the image to be processed yields the first encoded frame of the image, which can be implemented as follows: Step D1: Determine the geometric transformation model of the image to be processed based on the camera motion type.

[0097] Geometric transformation models can be translational, affine, similarity, etc. For example, when the camera motion is determined to be translational, a translational model can be used as the geometric transformation model; when the camera motion is determined to be pure rotational, 2D rotation can be used as the geometric transformation model. Based on a defined geometric transformation model, more accurate motion compensation prediction can be performed during the encoding process, thereby improving encoding efficiency.

[0098] Step D2: Encode the image to be processed based on the geometric transformation model to obtain the first encoded frame.

[0099] In the embodiments of this disclosure, by using motion vectors for outlier removal and camera motion type determination, the robustness of motion estimation and coding efficiency can be improved, while reducing the transmission bit rate.

[0100] In order to prioritize the transmission of image regions that are more important to the SLAM task under limited bandwidth, this embodiment of the disclosure further includes: identifying regions of interest and unreliable motion vector regions in the image to be processed based on motion vectors of multiple coding units.

[0101] Motion vectors contain rich spatial motion information. Motion vectors in different regions exhibit different characteristics in terms of amplitude, direction, and spatial consistency. These characteristics can be used to distinguish regions in an image that have different values ​​for solving SLAM ensembles. In implementation, based on motion vectors, regions with severe motion blur, occluded and deoccluded regions, and regions with unreliable motion vectors are identified. After identifying these three types of regions, different regions are combined according to the application objective to generate spatial filtering masks or Region of Interest (ROI) masks.

[0102] A region of interest (ROI) refers to a set of coding units that simultaneously meet the following conditions: (1) the region is not a region with severe motion blur; (2) the region is not an occluded or de-occluded region, meaning its motion vector is consistent with its spatial neighborhood and the matching cost is normal; and (3) the region is not a region with unreliable motion vectors. In other words, an ROI is an image region that is motion-significant, reliable, and has effective texture information, providing high-quality geometric constraints for cloud-based SLAM.

[0103] Unreliable motion vector regions refer to a set of coding units that meet one of the following conditions: (1) the region is an occluded or deoccluded region; (2) the region is a motion-blurred region. In other words, unreliable regions are image regions where motion vectors cannot truly reflect physical motion or lack texture information that can be used for feature extraction, which does not contribute to SLAM geometric solution and may even introduce errors.

[0104] Encoding the image to be processed yields the first encoded frame of the image, which can be implemented as follows: Step E1: Obtain the first quantization parameter corresponding to the region of interest and the second quantization parameter corresponding to the region of unreliable motion vector; the first quantization parameter is less than the second quantization parameter.

[0105] The quantization parameter (QP) is a core parameter in video encoding that controls the quantization step size and bitrate allocation. A smaller quantization parameter results in a smaller quantization step size, preserving more image quality, but also consuming more bitrate; a larger quantization parameter results in a larger quantization step size, leading to greater image quality loss, but also consuming less bitrate.

[0106] During implementation, a smaller first quantization parameter is assigned to the region of interest, resulting in a smaller quantization step size for that region. This allows for the preservation of more high-frequency detail information (such as edges and textures), which is beneficial for feature extraction and cross-frame matching by the cloud-side SLAM neural network. A larger second quantization parameter is assigned to regions with unreliable motion vectors, which performs strong quantization on these regions, significantly suppressing texture and noise and reducing the number of encoded bits.

[0107] Step E2: Encode the region of interest based on the first quantization parameter, and encode the unreliable region of motion vector based on the second quantization parameter to obtain the first coded frame.

[0108] The first and second quantization parameters are applied to the video encoding process to achieve differentiated bitrate allocation for different regions within the same frame.

[0109] During implementation, the region of interest mask and the unreliable region mask are input into the video encoder's application programming interface (API). When encoding each coding unit, the video encoder queries the region type of that unit based on the mask label and applies the corresponding encoding parameters.

[0110] For example, if the coding unit belongs to a region of interest (ROI), the first quantization parameter (low QP value) is applied to enable the encoder to perform finer quantization on that region, preserving more high-frequency detail information (such as edges and textures), which is beneficial for feature extraction and cross-frame matching in the cloud-based SLAM neural network. If the coding unit belongs to an unreliable region of motion vectors, the second quantization parameter (high QP value) is applied to enable the encoder to perform coarser quantization on that region, significantly suppressing texture and noise and reducing bitrate consumption. If the coding unit belongs to a normal region (i.e., neither ROI nor unreliable region), the baseline quantization parameter is applied to that region.

[0111] In this embodiment of the disclosure, by identifying regions of interest and unreliable regions based on motion vectors and assigning different quantization parameters to different regions, the encoding quality of key image regions is prioritized under limited transmission bandwidth, thereby ensuring the inference accuracy of cloud-based SLAM in key regions without significantly increasing the overall bitrate.

[0112] To effectively manage and utilize the inference results returned from the cloud on the device side, after sending the first encoded frame of the image to be processed to the cloud, the process also includes, for example... Figure 2 The steps shown are as follows: S201, store the image to be processed and its timestamp in a preset image set.

[0113] The preset image set is an image buffer queue maintained locally by the terminal device to store image frames that have been sent to the cloud and their corresponding timestamps, so as to align them with the inference results returned by the cloud later.

[0114] Understandably, the interpolated frames and their corresponding timestamps are also stored synchronously in the preset image set to provide more reliable historical data for subsequent optimization.

[0115] S202 receives the inference results of the feedforward SLAM neural network model sent from the cloud.

[0116] The inference results include one or more of the following: camera pose, depth map, and 3D point cloud. Each result has its own corresponding timestamp to be compared.

[0117] S203, align the timestamps to be compared in the inference results with the timestamps in the image set to associate the inference results with the corresponding image frames, and obtain the updated image set.

[0118] During implementation, based on the inference results sent from the cloud, the timestamps to be compared in the inference results are aligned with the timestamps in the image set. This allows for the search within a preset image set for image frames whose timestamps match the timestamps to be compared or are within a preset tolerance, thus obtaining an updated image set. It should be noted that if no image frame corresponding to the inference result exists in the preset image set, the inference result is discarded or marked as misaligned.

[0119] S204, if it is determined that extrapolation is needed, generate a new pose based on the updated image set.

[0120] In practice, when extrapolation is required, the poses of the most recent frames (e.g., frames 2-5) and their corresponding timestamps are extracted from the historical pose sequence of the updated image set, and a motion model is adaptively selected for extrapolation. The selectable motion models include: (1) a constant velocity model, which assumes constant linear and angular velocities; and (2) a uniform acceleration model, which assumes constant acceleration. In practice, any combination of one or more of the above motion models can be selected based on preset or adaptive methods.

[0121] Due to the inevitable latency introduced by cloud-side inference and edge-cloud network transmission, and the existence of jitter, packet loss, and brief interruptions in real-world networks, the pose obtained on the edge may lag behind the current rendering time. Therefore, pose extrapolation is necessary. Pose extrapolation refers to predicting the pose information at the current or future time based on historical pose information.

[0122] Determining whether extrapolation is needed can be done in ways including but not limited to the following: Method 1: Extrapolation determination based on point cloud timestamps. Specifically, when the inference result includes point cloud timestamps, the point cloud timestamps are compared with the latest pose timestamp in the updated image set; if the point cloud timestamp is earlier than the latest pose timestamp, and the first time difference between the point cloud timestamp and the latest pose timestamp is greater than a first preset threshold, it is determined that extrapolation is required.

[0123] When the point cloud timestamp is earlier than the latest pose timestamp, and the first time difference between the point cloud timestamp and the latest pose timestamp is greater than the first preset threshold, it indicates that the pose difference is large. Therefore, an extrapolation mechanism needs to be triggered to avoid visual inconsistency or jump problems.

[0124] It should be noted that the first preset threshold can be adaptively adjusted according to the statistical characteristics of the end-to-end transmission delay, and this disclosure does not impose any restrictions on it. For example, when the network condition is good, the latency is stable and the jitter is small, a smaller threshold can be set to trigger extrapolation more sensitively and maintain the real-time performance of the rendering pose; when the network condition fluctuates greatly, a larger threshold can be set to avoid unnecessary computational overhead and possible prediction noise caused by frequent extrapolation.

[0125] In this embodiment of the disclosure, by comparing the first time difference between the point cloud timestamp and the latest pose timestamp, and triggering extrapolation when the first time difference is greater than a first preset threshold, it is possible to ensure that the pose used for rendering and the scene geometric data remain aligned in time, thereby improving the realism and stability of visual fusion.

[0126] Method 2: Extrapolation determination based on predicted pose. Specifically, when the inference result includes the predicted pose, the target timestamp of the predicted pose output by the SLAM neural network model is compared with the current rendering time; if the target timestamp is earlier than the current rendering time, and the second time difference between the target timestamp and the current rendering time is greater than a second preset threshold, it is determined that extrapolation is required.

[0127] It should be noted that the second preset threshold can be set according to the rendering frame rate, and this disclosure does not impose any restrictions on it.

[0128] In this embodiment of the disclosure, by comparing the target timestamp of the predicted pose with the current rendering time, and triggering extrapolation when the second time difference is greater than the second preset threshold, the pose lag caused by transmission delay can be effectively compensated, so that the rendering position of the virtual object is kept synchronized with the actual scene, thereby significantly improving the real-time experience of edge rendering.

[0129] By using the two methods described above, the scene that needs to trigger pose extrapolation can be accurately identified, thereby initiating the extrapolation process in a timely manner and ensuring the continuity of rendering.

[0130] S205, based on the new pose, renders point cloud data sent from the cloud.

[0131] During implementation, the extrapolated new pose is used as the virtual camera pose for rendering. The point cloud data sent from the cloud is transformed from the world coordinate system to the camera coordinate system for point cloud rendering, and virtual objects are superimposed to complete the final rendering.

[0132] In this embodiment of the disclosure, by maintaining the image set on the edge and performing timestamp alignment, the accurate association between cloud inference results and local image frames is achieved; through the pose extrapolation mechanism, network latency is effectively compensated, ensuring the real-time performance and smoothness of rendering, and improving the user's spatial interaction experience.

[0133] In real-world network environments, communication quality may fluctuate. For example, when a device enters a network signal dead zone, Wi-Fi signal weakens, or network congestion leads to increased packet loss, data transmission between the device and the cloud may be interrupted or experience significant delays. To maintain the basic functionality and visual continuity of the application under these adverse conditions, embodiments of this disclosure also include a communication degradation processing strategy. Specifically, this includes the following steps: Step F1: If the communication quality with the cloud is lower than the expected quality, estimate the predicted camera pose of the image to be processed in the world coordinate system.

[0134] It should be noted that communication quality can be characterized by parameters such as network latency, packet loss rate, and signal strength. The desired quality can be set according to specific application requirements. For example, network outages or packet loss rates exceeding preset thresholds can be considered situations where communication quality is lower than the desired quality. When communication quality is lower than the desired quality, cloud-based inference results may not be delivered in a timely manner; in this case, the terminal device can activate its local estimation capabilities.

[0135] Step F2: Using the image to be processed as the background, the target object in the historical point cloud data is superimposed onto the image to be processed based on the predicted camera pose to obtain the rendering result.

[0136] Historical point cloud data refers to the last successfully transmitted 3D point cloud or 3D mesh model from the cloud, stored in the local cache on the device side. It contains the geometric and appearance information of the target object (such as vertex coordinates, color, texture coordinates, etc.). The target object refers to predefined digital content that needs to be presented in the augmented reality scene, including but not limited to navigation marks, information tags, virtual objects, etc.

[0137] During implementation, the edge device uses the currently acquired real-time image frame (i.e., the image to be processed) as the background. This image frame reflects the real physical environment at the current moment, thus ensuring the realism of the background. Then, based on the obtained predicted camera pose, the target object in the historical point cloud data is rendered.

[0138] In this embodiment of the disclosure, when the communication quality with the cloud is lower than expected, the edge device uses local pose estimation and image background-based rendering scheme to ensure the basic availability and visual consistency of the application under poor communication quality, avoid the application from completely failing due to communication interruption, and improve the robustness of the system.

[0139] Specifically, the predicted camera pose of the image to be processed in the world coordinate system can be estimated using one or a combination of the following methods: Method 1: Pose estimation based on lightweight vision.

[0140] This method includes the following steps: Step G1: Extract features from the image to be processed to obtain the first feature, and extract features from the reference frame in the preset image set to obtain the second feature; the reference frame is the image frame corresponding to the last known pose data sent from the cloud.

[0141] During implementation, the endpoint selects a reference frame from a preset image set. This reference frame is the image frame corresponding to the last successful pose transmission from the cloud. This reference frame contains the known camera pose in the world coordinate system (i.e., 6-DOF pose, including the rotation matrix R). ref Translation vector t refThis pose serves as the starting point for subsequent recursion. The edge device extracts features from both the image to be processed and the reference frame. Feature extraction methods include corner detection algorithms, binary descriptor algorithms, or blob detection algorithms. For resource-constrained edge devices, it is preferable to use computationally efficient features based on Accelerated Segment Test (FAST) or Oriented FAST and Rotated BRIEF (ORB) features to balance accuracy and power consumption. It should be noted that since this step is triggered when the communication quality in the cloud is lower than expected, and the edge hardware capabilities are limited, the complexity of feature extraction should not be too high to ensure that a sufficient number of stable feature points are extracted for subsequent matching while meeting real-time requirements.

[0142] Step G2 involves performing feature matching based on the first and second features to obtain the correspondence between key points between the image to be processed and the reference frame.

[0143] Keypoint correspondence refers to the correspondence between the pixel positions of the same 3D spatial point in the image to be processed and the reference frame. That is, for the same physical point in space, its pixel coordinates (u1, v1) in the image to be processed and its pixel coordinates (u2, v2) in the reference frame form a set of correspondences. By establishing a sufficient number of such correspondences, the relative pose transformation matrix between the two frames can be further solved.

[0144] Step G3: Based on the correspondence of key points, determine the relative motion of the image to be processed relative to the reference frame.

[0145] During implementation, based on the obtained keypoint correspondence, the epipolar geometry method is used to recover the relative pose transformation between two frames, and the relative rotation matrix R between the image to be processed and the reference frame is determined. rel and relative translation vector T rel .

[0146] Step G4: Based on the relative motion and the known pose data of the reference frame, determine the predicted camera pose of the image to be processed in the world coordinate system.

[0147] During implementation, the relative rotation matrix R obtained in step F3 is used... rel and relative translation vector T rel The predicted camera pose is obtained by superimposing it onto the known camera pose of the reference frame.

[0148] In this embodiment, by performing lightweight visual computation on the device side, the predicted camera pose of the image to be processed is determined based on feature matching and relative motion estimation between the current frame and the last reference frame sent from the cloud. This maintains the continuity of pose output even without cloud inference results, avoiding rendering stagnation or screen stuttering due to communication interruptions. Furthermore, this method can continuously output a predicted pose with a certain accuracy during communication interruptions. When communication is restored, the device side can realign based on the accurate pose returned from the cloud, effectively avoiding the impact of pose jumps on the user experience.

[0149] Method 2: Pose estimation based on inertial measurement unit (IMU) data integration. This includes the following steps: Step H1: Obtain the measurement data from the inertial measurement unit and the known camera pose of the reference frame.

[0150] During implementation, the device reads measurement data from its built-in IMU sensors. The measurement data includes triaxial angular velocity (gyroscope measurement, unit: rad / s) and triaxial acceleration (accelerometer measurement, unit: m / s²). 2 Simultaneously, the edge device obtains the known camera pose of the reference frame from a preset image set.

[0151] Step H2: Using the known camera pose as the starting point for integration, integrate the angular velocity in the measurement data to obtain the relative rotation.

[0152] During implementation, the known camera pose of the reference frame is used as the starting point for integration, and the angular velocity in the measurement data is integrated. For example, quaternions or rotation matrices can be used to recursively obtain the relative rotation change from the reference frame time to the current time.

[0153] Step H3: Transform the acceleration in the measurement data to the world coordinate system to obtain the motion acceleration.

[0154] During implementation, the accelerometer measurements need to be converted to the world coordinate system and the gravitational component needs to be subtracted to obtain the actual acceleration.

[0155] To suppress high-frequency noise from the accelerometer, the original acceleration measurement can be low-pass filtered before converting the acceleration to the world coordinate system to remove high-frequency vibration noise while retaining low-frequency motion components.

[0156] Step H4: Integrate the acceleration to obtain the relative displacement.

[0157] In practice, the acceleration is double-integrated to obtain the relative displacement from the reference frame to the current time. Specifically, the first integration yields the velocity; the second integration yields the position shift, i.e., the relative displacement.

[0158] To suppress integral drift, the velocity can be forced to zero when the device is detected to be stationary (e.g., by determining the variance of the accelerometer), or a threshold can be set for the acceleration measurement level to cut off the accumulation path of drift and significantly improve the usability of pure inertial recursion on medium and long time scales.

[0159] Step H5: The relative rotation and relative displacement are superimposed on the known camera pose to obtain the predicted camera pose.

[0160] During implementation, the relative rotation and relative displacement are combined to obtain the transformation matrix T. imu_rel Then, this transformation matrix is ​​superimposed on the known camera pose transformation matrix of the reference frame.

[0161] In this embodiment, the predicted camera pose is calculated on the edge using pure inertial recursion, thereby maintaining the continuity of pose output without relying on image input. Pure inertial recursion is unaffected by visual factors such as changes in illumination, missing textures, or motion blur, and can work stably under any visual conditions. Moreover, the calculation process only involves matrix multiplication and integration, resulting in low computational overhead. It can run continuously on the edge without GPU acceleration and can serve as a complementary solution in scenarios where visual odometry fails.

[0162] Figure 3 This is a schematic diagram illustrating the implementation process of an edge-cloud collaborative SLAM method for spatial intelligence according to an embodiment of the present disclosure. The edge-cloud collaborative SLAM method for spatial intelligence in this example can be implemented through the following steps, specifically including: S301, acquire the image to be processed acquired by the image acquisition device.

[0163] S302, acquire the motion vectors of multiple coding units of the image to be processed.

[0164] S303, based on the motion vectors of multiple coding units, determines the cumulative motion of the image to be processed relative to the target transmitted frame; the target transmitted frame is the frame image that is closest in time to the image to be processed in the frame sequence transmitted to the cloud.

[0165] S304 compares the cumulative exercise volume with the preset exercise volume range.

[0166] S305: If the cumulative motion volume is lower than the minimum value of the preset motion volume range, filter out the image to be processed.

[0167] S306, when the cumulative motion amount is within the preset motion amount range, the image to be processed is encoded to obtain the first encoded frame of the image to be processed.

[0168] S307, when the cumulative motion amount is higher than the highest value of the preset motion amount range, an interpolation operation is performed based on the image to be processed and the target transmitted frame to obtain at least one interpolated frame; the image to be processed is encoded to obtain a first encoded frame of the image to be processed; and the at least one interpolated frame is encoded to obtain a second encoded frame of the at least one interpolated frame.

[0169] S308 sends the first encoded frame of the image to be processed to the cloud so that the cloud can process the image to be processed through a feedforward SLAM neural network model.

[0170] S309, the image to be processed and its timestamp are associated and stored in a preset image set.

[0171] S310, receive the inference results of the feedforward real-time localization and mapping neural network model sent by the cloud.

[0172] S311, Align the timestamp to be compared in the inference result with the timestamp in the image set to associate the inference result with the corresponding image frame, and obtain the updated image set.

[0173] S312, if it is determined that extrapolation is needed, a new pose is generated based on the updated image set.

[0174] S313, Based on the new pose, render the point cloud data sent from the cloud.

[0175] Based on the same technical concept, embodiments of this disclosure provide an edge-cloud collaborative SLAM system for spatial intelligence, such as... Figure 4 As shown, the system includes: Terminal device 410 is configured to acquire an image to be processed acquired by an image acquisition device; acquire motion vectors of multiple coding units of the image to be processed; determine the cumulative motion of the image to be processed relative to a target transmitted frame based on the motion vectors of the multiple coding units; the target transmitted frame is the frame image that is closest in time to the image to be processed in the frame sequence transmitted to the cloud; compare the cumulative motion with a preset motion range; if the cumulative motion is lower than the minimum value of the preset motion range, filter out the image to be processed; if the cumulative motion is within the preset motion range, encode the image to be processed to obtain a first encoded frame of the image to be processed; and send the first encoded frame of the image to be processed to the cloud. Cloud 420 is used to process the image to be processed through a feedforward real-time localization and mapping neural network model to obtain inference results, and send the inference results to the terminal device.

[0176] Based on the same technical concept, this disclosure also provides an edge-cloud collaborative SLAM device 500 for spatial intelligence, such as... Figure 5 As shown, the device includes: The acquisition module 501 is used to acquire the image to be processed acquired by the image acquisition device; and to acquire the motion vectors of multiple coding units of the image to be processed. The motion determination module 502 is used to determine the cumulative motion of the image to be processed relative to the target transmitted frame based on the motion vectors of the plurality of coding units; the target transmitted frame is the frame image that is closest in time to the image to be processed in the frame sequence transmitted to the cloud. Comparison module 503 is used to compare the cumulative exercise volume with a preset exercise volume range; The filtering module 504 is used to filter out the image to be processed when the cumulative motion amount is lower than the minimum value of the preset motion amount range. The encoding module 505 is used to encode the image to be processed when the cumulative motion amount is within the preset motion amount range, so as to obtain the first encoded frame of the image to be processed. The sending module is used to send the first encoded frame of the image to be processed to the cloud, so that the cloud can process the image to be processed through a feedforward real-time localization and mapping neural network model.

[0177] The device further includes: An interpolation module is used to perform an interpolation operation based on the image to be processed and the target transmitted frame when the cumulative motion amount is higher than the highest value of the preset motion amount range, so as to obtain at least one interpolated frame. The encoding module is used to encode the image to be processed to obtain a first encoded frame of the image to be processed; and to encode the at least one interpolated frame to obtain a second encoded frame of the at least one interpolated frame. The sending module is used to send the first encoded frame and the second encoded frame to the cloud.

[0178] The device further includes: The loop control module is used to, after filtering out the image to be processed, take the next frame of the image to be processed acquired by the image acquisition device as the new image to be processed, and return to the step of obtaining the motion vectors of the multiple encoding units of the image to be processed.

[0179] The device further includes: The preprocessing module is used to remove outliers from the multiple coding units based on their motion vectors, thereby obtaining optimized coding units. A motion type recognition module is used to determine the camera motion type for the image to be processed based on the motion vector of the optimized encoding unit. The encoding module includes: A geometric transformation model determination unit is used to determine the geometric transformation model of the image to be processed based on the camera motion type; The encoding unit is used to encode the image to be processed based on the geometric transformation model to obtain the first encoded frame.

[0180] The device further includes: The region identification module is used to identify regions of interest and regions with unreliable motion vectors in the image to be processed based on the motion vectors of the multiple coding units. The encoding module includes: A quantization parameter acquisition unit is used to acquire a first quantization parameter corresponding to the region of interest and a second quantization parameter corresponding to the unreliable region of the motion vector; the first quantization parameter is less than the second quantization parameter. The encoding unit is used to encode the region of interest based on the first quantization parameter and to encode the unreliable region of the motion vector based on the second quantization parameter to obtain the first encoded frame.

[0181] The device further includes: The image storage module is used to associate and store the image to be processed and its timestamp in a preset image set. The result receiving module is used to receive the inference results of the feedforward real-time localization and mapping neural network model sent by the cloud. The alignment module is used to align the timestamp to be compared in the inference result with the timestamp in the image set, so as to associate the inference result with the corresponding image frame and obtain the updated image set. The pose generation module is used to generate a new pose based on the updated image set when it is determined that extrapolation is required; The rendering module is used to render the point cloud data sent from the cloud based on the new pose.

[0182] The device further includes an extrapolation determination module, used for: If the inference result includes a point cloud timestamp, compare the point cloud timestamp with the latest pose timestamp in the updated image set; If the point cloud timestamp is earlier than the latest pose timestamp, and the first time difference between the point cloud timestamp and the latest pose timestamp is greater than a first preset threshold, it is determined that extrapolation is required.

[0183] The device further includes an extrapolation determination module, used for: If the inference result includes a predicted pose, the target timestamp of the predicted pose is compared with the current rendering time. If the target timestamp is earlier than the current rendering time, and the second time difference between the target timestamp and the current rendering time is greater than a second preset threshold, it is determined that extrapolation is required.

[0184] The device further includes: The degradation processing module is used to estimate the predicted camera pose of the image to be processed in the world coordinate system when the communication quality with the cloud is lower than the expected quality. The rendering module is used to use the image to be processed as a background, and to overlay the target object in the historical point cloud data onto the image to be processed based on the predicted camera pose to obtain the rendering result.

[0185] The downgrade processing module includes: The feature extraction unit is used to extract features from the image to be processed to obtain a first feature, and to extract features from a reference frame in the preset image set to obtain a second feature; the reference frame is the image frame corresponding to the last known pose data sent from the cloud. A feature matching unit is used to perform feature matching based on the first feature and the second feature to obtain the key point correspondence between the image to be processed and the reference frame; A relative motion determination unit is used to determine the relative motion of the image to be processed relative to the reference frame based on the correspondence of the key points; The first predicted pose determination unit is used to determine the predicted camera pose of the image to be processed in the world coordinate system based on the relative motion and the known pose data of the reference frame.

[0186] The downgrade processing module includes: The data acquisition unit is used to acquire measurement data from the inertial measurement unit and the known camera pose of the reference frame; The relative rotation determination unit is used to integrate the angular velocity in the measurement data using the known camera pose as the integration starting point to obtain the relative rotation. The motion acceleration determination unit is used to transform the acceleration in the measurement data to the world coordinate system to obtain the motion acceleration. A relative displacement determination unit is used to integrate the motion acceleration to obtain the relative displacement; The second predicted pose determination unit is used to superimpose the relative rotation and the relative displacement onto the known camera pose to obtain the predicted camera pose.

[0187] The device further includes the interpolation module, comprising: The quantity determination unit is used to determine the number N of interpolation frames based on the ratio of the cumulative motion amount to the highest value of the preset motion amount interval; N is an integer greater than or equal to 1, and N is proportional to the ratio. An interpolation coefficient determination unit is used to determine the interpolation coefficients of the kth interpolation frame among N interpolation frames; the kth interpolation frame is any frame among the N interpolation frames; wherein the interpolation coefficients are proportional to k and inversely proportional to N; An intermediate motion vector determination unit is used to linearly scale the motion vectors of the plurality of coding units based on the interpolation coefficients to obtain the intermediate motion vectors of each coding unit in the image to be processed relative to the target transmitted frame. An interpolation frame generation unit is used to perform linear motion compensation displacement on the pixels of the corresponding coding unit in the target transmitted frame based on the intermediate motion vector, and generate the k-th interpolation frame.

[0188] The specific functions and examples of each module and submodule of the apparatus in this disclosure can be found in the relevant descriptions of the corresponding steps in the above method embodiments, and will not be repeated here.

[0189] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0190] Figure 6 This is a structural block diagram of an electronic device according to an embodiment of the present disclosure. Figure 6 As shown, the electronic device includes a memory 610 and a processor 620. The memory 610 stores a computer program that can run on the processor 620. There can be one or more memories 610 and processors 620. The memory 610 can store one or more computer programs, which, when executed by the electronic device, cause the electronic device to perform the methods provided in the above-described method embodiments. The electronic device may also include a communication interface 630 for communicating with external devices and performing data exchange and transmission.

[0191] If the memory 610, processor 620, and communication interface 630 are implemented independently, they can be interconnected via a bus to communicate with each other. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 6 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0192] Optionally, in a specific implementation, if the memory 610, processor 620, and communication interface 630 are integrated on a single chip, then the memory 610, processor 620, and communication interface 630 can communicate with each other through an internal interface.

[0193] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting Advanced Reduced Instruction Set Machines (ARM) architecture.

[0194] Further, optionally, the aforementioned memory may include read-only memory and random access memory, and may also include non-volatile random access memory. The memory may be volatile or non-volatile, or may include both. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache. Many forms of RAM are available by way of example, but not limitation. Examples include Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct RAMBUS RAM (DR RAM).

[0195] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this disclosure are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line, DSL) or wireless (e.g., infrared, Bluetooth, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer, or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., Digital Versatile Discs (DVDs)), or semiconductor media (e.g., Solid State Disks (SSDs)). It is worth noting that the computer-readable storage media mentioned in this disclosure can be non-volatile storage media; in other words, it can be non-transient storage media.

[0196] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0197] In the description of the embodiments of this disclosure, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0198] In the description of the embodiments disclosed herein, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone.

[0199] In the description of embodiments of this disclosure, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of this disclosure, unless otherwise stated, "a plurality of" means two or more.

[0200] The above description is merely an exemplary embodiment of this disclosure and is not intended to limit this disclosure. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the protection scope of this disclosure.

Claims

1. A method for real-time localization and mapping via edge-cloud collaboration for spatial intelligence, comprising: Acquire the image to be processed from the image acquisition device; Obtain the motion vectors of multiple coding units of the image to be processed; Based on the motion vectors of the multiple coding units, the cumulative motion of the image to be processed relative to the target transmitted frame is determined; the target transmitted frame is the frame image that is closest to the image to be processed in time in the frame sequence transmitted to the cloud. Compare the cumulative exercise volume with a preset exercise volume range; If the cumulative motion amount is lower than the lowest value of the preset motion amount range, the image to be processed is filtered out. When the cumulative motion amount is within the preset motion amount range, the image to be processed is encoded to obtain the first encoded frame of the image to be processed. The first encoded frame of the image to be processed is sent to the cloud so that the cloud processes the image to be processed through a feedforward real-time localization and mapping neural network model.

2. The method according to claim 1, further comprising: When the cumulative motion amount is higher than the highest value of the preset motion amount range, an interpolation operation is performed based on the image to be processed and the target transmitted frame to obtain at least one interpolated frame. After encoding the image to be processed, the first encoded frame of the image to be processed is obtained; as well as, After encoding the at least one interpolated frame, a second encoded frame of the at least one interpolated frame is obtained; The first encoded frame and the second encoded frame are sent to the cloud.

3. The method according to claim 1, further comprising, after filtering out the image to be processed: The next frame of the image to be processed, acquired by the image acquisition device, is used as the new image to be processed, and the process returns to the step of obtaining the motion vectors of the multiple coding units of the image to be processed.

4. The method according to any one of claims 1-3, further comprising: Based on the motion vectors of the multiple coding units, outliers in the multiple coding units are removed to obtain optimized coding units; Based on the motion vector of the optimized coding unit, the camera motion type for the image to be processed is determined; The step of encoding the image to be processed to obtain the first encoded frame of the image to be processed includes: The geometric transformation model of the image to be processed is determined based on the camera motion type; The image to be processed is encoded based on the geometric transformation model to obtain the first encoded frame.

5. The method according to claim 1, further comprising: Based on the motion vectors of the multiple coding units, the region of interest and the region of unreliable motion vectors in the image to be processed are identified. The step of encoding the image to be processed to obtain the first encoded frame of the image to be processed includes: Obtain the first quantization parameter corresponding to the region of interest, and the second quantization parameter corresponding to the unreliable region of the motion vector; the first quantization parameter is less than the second quantization parameter; The region of interest is encoded based on the first quantization parameter, and the unreliable region of the motion vector is encoded based on the second quantization parameter to obtain the first encoded frame.

6. The method according to claim 1, further comprising, after sending the first encoded frame of the image to be processed to the cloud: The image to be processed and its timestamp are associated and stored in a preset image set; Receive the inference results of the feedforward real-time localization and mapping neural network model sent from the cloud; Align the timestamps to be compared in the inference results with the timestamps in the image set to associate the inference results with the corresponding image frames, thereby obtaining the updated image set; If extrapolation is required, a new pose is generated based on the updated image set; Based on the new pose, render the point cloud data sent from the cloud.

7. The method according to claim 6, further comprising: If the inference result includes a point cloud timestamp, compare the point cloud timestamp with the latest pose timestamp in the updated image set; If the point cloud timestamp is earlier than the latest pose timestamp, and the first time difference between the point cloud timestamp and the latest pose timestamp is greater than a first preset threshold, it is determined that extrapolation is required.

8. The method according to claim 6, further comprising: If the inference result includes a predicted pose, the target timestamp of the predicted pose is compared with the current rendering time. If the target timestamp is earlier than the current rendering time, and the second time difference between the target timestamp and the current rendering time is greater than a second preset threshold, it is determined that extrapolation is required.

9. The method according to any one of claims 6-8, further comprising: When the communication quality with the cloud is lower than the expected quality, the predicted camera pose of the image to be processed in the world coordinate system is estimated. Using the image to be processed as a background, the target object in the historical point cloud data is superimposed onto the image to be processed based on the predicted camera pose to obtain the rendering result.

10. The method according to claim 9, wherein, The step of estimating the predicted camera pose of the image to be processed in the world coordinate system includes: The image to be processed is subjected to feature extraction to obtain a first feature, and a reference frame in the preset image set is subjected to feature extraction to obtain a second feature; the reference frame is the image frame corresponding to the last known pose data sent by the cloud. Based on the first feature and the second feature, feature matching is performed to obtain the key point correspondence between the image to be processed and the reference frame; Based on the correspondence of the key points, the relative motion of the image to be processed relative to the reference frame is determined; Based on the relative motion and the known pose data of the reference frame, the predicted camera pose of the image to be processed in the world coordinate system is determined.

11. The method according to claim 9, wherein, The step of estimating the predicted camera pose of the image to be processed in the world coordinate system includes: Acquire measurement data from the inertial measurement unit and the known camera pose of the reference frame; Using the known camera pose as the starting point for integration, the angular velocity in the measurement data is integrated to obtain the relative rotation; The acceleration in the measured data is transformed to the world coordinate system to obtain the motion acceleration; Integrating the acceleration, the relative displacement is obtained; The relative rotation and the relative displacement are superimposed on the known camera pose to obtain the predicted camera pose.

12. The method according to claim 2, wherein, The interpolation operation based on the image to be processed and the target transmitted frame to obtain at least one interpolated frame includes: The number of interpolation frames N is determined based on the ratio of the cumulative motion amount to the highest value of the preset motion amount range; N is an integer greater than or equal to 1, and N is proportional to the ratio. For the k-th interpolated frame among N interpolated frames, determine the interpolation coefficients of the k-th interpolated frame; the k-th interpolated frame is any frame among the N interpolated frames; wherein the interpolation coefficients are proportional to k and inversely proportional to N; Based on the interpolation coefficients, the motion vectors of the plurality of coding units are linearly scaled to obtain the intermediate motion vectors of each coding unit in the image to be processed relative to the target transmitted frame; Based on the intermediate motion vector, the pixels of the corresponding coding unit in the target transmitted frame are linearly motion compensated and displaced to generate the k-th interpolated frame.

13. A real-time positioning and mapping system for spatial intelligence, comprising: Terminal equipment, used to acquire images to be processed acquired by an image acquisition device; Obtain the motion vectors of multiple coding units of the image to be processed; Based on the motion vectors of the multiple coding units, the cumulative motion of the image to be processed relative to the target transmitted frame is determined; the target transmitted frame is the frame image that is closest to the image to be processed in time in the frame sequence transmitted to the cloud; the cumulative motion is compared with a preset motion range; if the cumulative motion is lower than the minimum value of the preset motion range, the image to be processed is filtered out; if the cumulative motion is within the preset motion range, the image to be processed is encoded to obtain the first encoded frame of the image to be processed. Send the first encoded frame of the image to be processed to the cloud; In the cloud, the image to be processed is processed through a feedforward real-time localization and mapping neural network model to obtain inference results, and the inference results are sent to the terminal device.

14. A device for real-time positioning and mapping via edge-cloud collaboration for spatial intelligence, comprising: The acquisition module is used to acquire the image to be processed acquired by the image acquisition device; Obtain the motion vectors of multiple coding units of the image to be processed; The motion determination module is used to determine the cumulative motion of the image to be processed relative to the target transmitted frame based on the motion vectors of the plurality of coding units. The target transmitted frame is the frame that is closest in time to the image to be processed in the frame sequence transmitted to the cloud; The comparison module is used to compare the cumulative exercise volume with a preset exercise volume range; A filtering module is used to filter out the image to be processed when the cumulative motion amount is lower than the minimum value of the preset motion amount range; The encoding module is used to encode the image to be processed when the cumulative motion amount is within the preset motion amount range, so as to obtain the first encoded frame of the image to be processed. The sending module is used to send the first encoded frame of the image to be processed to the cloud, so that the cloud can process the image to be processed through a feedforward real-time localization and mapping neural network model.

15. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-12.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-12.

17. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-12.

Images