Robot control end-to-end time delay measurement method and device and electronic equipment

Abstract

The invention relates to the technical field of time delay measurement, and provides a robot control end-to-end time delay measurement method and device and electronic equipment. The method comprises the following steps: acquiring a trigger timestamp of a robot motion instruction; based on a candidate displacement composite search strategy and a cost function early stop mechanism, performing global motion estimation on a monitoring video stream corresponding to the robot to obtain a global timestamp of global motion of a monitoring image in the monitoring video stream; the candidate displacement composite search strategy is used for adjusting a search sequence of candidate displacement between adjacent monitoring images based on motion space-time continuity; under the condition that the candidate displacement triggers a cost function early stop mechanism, the cost function early stop mechanism is used for terminating calculation of a cost function of the candidate displacement in advance; and determining an end-to-end time delay corresponding to the robot based on the global timestamp and the trigger timestamp. According to the embodiment of the invention, the end-to-end time delay of a complete link can be measured, and the accuracy and real-time performance of time delay measurement are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of delay measurement, in particular to a robot control end-to-end delay measurement method, device and electronic equipment. BACKGROUND

[0002] Among the many performance indicators of embodied intelligent robots, the real-time performance of control, that is, the delay time from when the user issues an instruction through an application (App) to when the robot presents the corresponding action in the App monitoring screen, is one of the most critical factors affecting user experience.

[0003] Currently, one traditional delay measurement method in the industry is the "timestamp comparison method". When a robot captures a video frame, a timestamp is attached to each frame of image data. When the App receives and prepares to display the video frame, the embedded timestamp is read and compared with the current system time of the App. The difference between the two is considered to be the delay.

[0004] However, the delay measurement link of this method is incomplete, and there is a dynamically changing clock bias between the robot system clock and the system clock of the App, resulting in low accuracy and real-time performance of the delay measurement result. Therefore, how to improve the accuracy and real-time performance of the robot end-to-end delay measurement is a problem that needs to be solved. SUMMARY

[0005] The embodiments of the present application provide a robot control end-to-end delay measurement method, device and electronic equipment to solve the problem of incomplete robot delay measurement link, dynamically changing clock bias between the robot system clock and the system clock of the App, and low accuracy and real-time performance of the delay measurement result in the prior art.

[0006] In a first aspect, the embodiments of the present application provide a robot control end-to-end delay measurement method, comprising the following steps.

[0007] Obtain the trigger timestamp of the robot motion instruction.

[0008] Based on a candidate displacement composite search strategy and an early stopping mechanism of a cost function, perform global motion estimation on the monitoring video stream corresponding to the robot to obtain a global timestamp of global motion of the monitoring images in the monitoring video stream. The candidate displacement composite search strategy is used to adjust the search order of the candidate displacement between adjacent monitoring images based on motion spatiotemporal continuity. In the case that the candidate displacement triggers the early stopping mechanism of the cost function, the early stopping mechanism of the cost function is used to terminate the calculation of the cost function of the candidate displacement in advance.

[0009] determine an end-to-end latency corresponding to the robot based on the global timestamp and the trigger timestamp.

[0010] In one embodiment, the global motion estimation of the monitoring video stream corresponding to the robot based on the candidate displacement composite search strategy and the early stopping mechanism of the cost function comprises: traversing the monitoring video stream corresponding to the robot to obtain a current frame monitoring image, a previous frame monitoring image and a previous frame best motion vector corresponding to the robot; determining a plurality of candidate displacements of the current frame based on the previous frame best motion vector and the candidate displacement composite search strategy; determining a current frame best motion vector and a global minimum cost based on the current frame monitoring image, the previous frame monitoring image, all candidate displacements and the early stopping mechanism of the cost function; performing global motion estimation based on the current frame best motion vector and the global minimum cost to obtain a global motion estimation result corresponding to the current frame; in a case where the global motion estimation result represents that the monitoring image corresponding to the robot has global motion, stopping the traversal and determining a timestamp corresponding to the current frame as the global timestamp of the global motion of the monitoring image corresponding to the robot.

[0011] In one embodiment, the candidate displacement composite search strategy comprises at least a neighborhood priority search strategy, an axial constraint search strategy and an extended ring traversal search strategy. the neighborhood priority search strategy is used to search the candidate displacements of the current frame in order of distance from the previous frame best motion vector from small to large; the axial constraint search strategy is used to constrain the search step length of the current frame in a second motion direction based on a first motion direction of the previous frame; the first motion direction is perpendicular to the second motion direction; the extended ring traversal search strategy is used to search the candidate displacements of the current frame in order of Manhattan distance from the origin from small to large; the remaining displacement is the displacement remaining in the search space after performing the neighborhood priority search strategy.

[0012] In one embodiment, the determination of the current frame best motion vector and the global minimum cost based on the current frame monitoring image, the previous frame monitoring image, all candidate displacements and the early stopping mechanism of the cost function comprises: S1, determining a current frame grayscale image corresponding to the current frame monitoring image and a previous frame grayscale image corresponding to the previous frame monitoring image; S2, traverse all candidate displacements, determine a final cost accumulation sum of the current candidate displacement that does not trigger the early stop mechanism of the cost function based on the current frame gray image, the previous frame gray image and the current candidate displacement; S3, determine the final cost accumulation sum as a new current minimum cost, and determine a next candidate displacement as a new current candidate displacement; S4, repeat steps S2 and S3 until all candidate displacements are traversed, and determine a final current minimum cost as a final minimum cost; S5, determine the current frame best motion vector and the global minimum cost based on the final minimum cost.

[0013] In an embodiment, the determining the final cost accumulation sum of the current candidate displacement that does not trigger the early stop mechanism of the cost function based on the current frame gray image, the previous frame gray image and the current candidate displacement comprises: S21, determine an overlapping area of the current frame gray image and the previous frame gray image based on the current candidate displacement; S22, traverse each pixel in the overlapping area, determine a local cost accumulation sum corresponding to the current candidate displacement based on a current pixel value of a target pixel in the current frame gray image, a previous pixel value of the target pixel in the previous frame gray image and the current candidate displacement; the target pixel includes a current pixel and all historical pixels before the current pixel; S23, in a case that the local cost accumulation sum is greater than or equal to the current minimum cost, trigger the early stop mechanism of the cost function to terminate the calculation of the current candidate displacement; determine the next candidate displacement as a new current candidate displacement, and repeat steps S2 and S5; S24, in a case that the local cost accumulation sum is less than the current minimum cost, add a next pixel to the target pixel, and repeat steps S22 and S24 based on the new target pixel until all pixels in the overlapping area are traversed, and determine a final local cost accumulation sum as the final cost accumulation sum of the current candidate displacement that does not trigger the early stop mechanism of the cost function.

[0014] In an embodiment, the determining the current frame best motion vector and the global minimum cost based on the final minimum cost comprises: determining the global minimum cost based on the final minimum cost and a total number of pixels in the overlapping area; determining a candidate displacement corresponding to the final minimum cost as the current frame best motion vector.

[0015] In one embodiment, the determining the current frame gray image corresponding to the current frame monitoring image and the previous frame gray image corresponding to the previous frame monitoring image comprises: Based on a preset down-sampling size, the current frame monitoring image and the previous frame monitoring image are down-sampled respectively to obtain a current frame down-sampled image corresponding to the current frame monitoring image and a previous frame down-sampled image corresponding to the previous frame monitoring image; The current frame down-sampled image and the previous frame down-sampled image are respectively grayed to obtain a current frame gray image corresponding to the current frame down-sampled image and a previous frame gray image corresponding to the previous frame down-sampled image.

[0016] In one embodiment, the global motion estimation based on the current frame optimal motion vector and the global minimum cost comprises: Based on the non-displacement global cost and the global minimum cost, a cost reduction rate is determined; Based on the non-displacement global cost, the cost reduction rate, the global minimum cost and the length of the current frame optimal motion vector, a global motion estimation result corresponding to the current frame is determined.

[0017] In one embodiment, the global motion estimation result is determined as that the monitoring image corresponding to the robot has global motion, comprising: In the case that the non-displacement global cost is greater than or equal to a noise threshold, the cost reduction rate is greater than a first preset threshold, the global minimum cost is greater than 0 and the length of the current frame optimal motion vector is greater than a second preset threshold, the global motion estimation result is determined as that the monitoring image corresponding to the robot has global motion.

[0018] In a second aspect, the embodiments of the present application provide a robot control end-to-end delay measurement device, comprising the following modules.

[0019] The acquisition module is configured to acquire a trigger timestamp of a robot motion instruction; The global motion estimation module is configured to perform global motion estimation on a monitoring video stream corresponding to a robot based on a candidate displacement composite search strategy and a cost function early stop mechanism to obtain a global timestamp of global motion of a monitoring image in the monitoring video stream; the candidate displacement composite search strategy is configured to adjust a search order of candidate displacements between adjacent monitoring images based on motion spatiotemporal continuity; in the case that a candidate displacement triggers the cost function early stop mechanism, the cost function early stop mechanism is configured to terminate calculation of a cost function of the candidate displacement in advance; The determination module is configured to determine an end-to-end delay corresponding to the robot based on the global timestamp and the trigger timestamp.

[0020] In a third aspect, an electronic device is provided, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the robot control end-to-end latency measurement method according to the first aspect when executing the computer program.

[0021] In a fourth aspect, a non-transitory computer-readable storage medium is provided, which stores a computer program, and the computer program is executable on a processor to implement the robot control end-to-end latency measurement method according to the first aspect.

[0022] In a fifth aspect, a computer program product is provided, which includes a computer program, and the computer program is executable on a processor to implement the robot control end-to-end latency measurement method according to the first aspect.

[0023] The robot control end-to-end latency measurement method, device, and electronic device provided by the embodiments of the present application can obtain a trigger timestamp of a robot motion instruction, and then perform global motion estimation on a monitoring video stream of the robot according to a candidate displacement composite search strategy for adjusting a search order of candidate displacements between adjacent monitoring images based on motion spatiotemporal continuity, and a cost function early termination mechanism for early termination of calculation of a cost function of the candidate displacements, to obtain a global timestamp of global motion of a monitoring image corresponding to the robot, and determine an end-to-end latency of the robot according to the global timestamp and the trigger timestamp. In the embodiments of the present application, the measured is an end-to-end latency of a complete link corresponding to a client that issues a robot motion instruction to a monitoring image of the client in which global motion occurs, and both the trigger timestamp and the global timestamp are determined based on a system clock of the same client, thereby avoiding dynamic clock deviation caused by different system clocks of different devices, and improving the accuracy of end-to-end latency measurement. Meanwhile, the candidate displacement composite search strategy and the cost function early termination mechanism are optimized cooperatively, which greatly reduces the calculation amount of global motion estimation, improves the calculation efficiency of latency measurement, and thus improves the real-time performance of end-to-end latency measurement. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0025] Figure 1 FIG. 1 is a flowchart of a robot control end-to-end latency measurement method provided by the embodiments of the present application.

[0026] Figure 2is a schematic diagram of a current frame monitoring image before and after down-sampling provided by an embodiment of the present application.

[0027] Figure 3 is a schematic diagram of a previous frame monitoring image before and after down-sampling provided by an embodiment of the present application.

[0028] Figure 4 is a structural schematic diagram of a robot control end-to-end latency measurement device provided by an embodiment of the present application.

[0029] Figure 5 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0030] To make the objectives, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described below in detail with reference to the drawings in the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0031] In view of the problem in the prior art that the robot latency measurement link is incomplete, there is a dynamic clock deviation between the system clock of the robot and the system clock of the App end, and the accuracy and real-time performance of the latency measurement result is low, an embodiment of the present application provides a robot control end-to-end latency measurement method, Figure 1 is a flowchart of a robot control end-to-end latency measurement method provided by an embodiment of the present application, as shown in Figure 1 The method comprises the following steps 110 to 130.

[0032] Step 110, obtaining a trigger timestamp of a robot motion instruction.

[0033] Specifically, the robot motion instruction is an instruction for making the robot produce physical displacement or attitude change triggered by a user through a monitoring interface of an application (Application, App) of an electronic device (i.e., a client), for example, instructions such as rotating left, advancing, retreating, etc. After the user triggers the robot motion instruction, the electronic device obtains a trigger timestamp of triggering the robot motion instruction through a system clock. The trigger timestamp accurately marks the starting time of the user's intention conversion into a specific instruction, and is also the starting point of the entire latency measurement link.

[0034] In step 120, global motion estimation is performed on the monitoring video stream corresponding to the robot based on a candidate displacement composite search strategy and a cost function early stopping mechanism, to obtain a global timestamp of global motion of a monitoring image in the monitoring video stream; the candidate displacement composite search strategy is used to adjust the search order of candidate displacements between adjacent monitoring images based on motion spatiotemporal continuity; in the case that a candidate displacement triggers the cost function early stopping mechanism, the cost function early stopping mechanism is used to terminate the calculation of the cost function of the candidate displacement in advance.

[0035] The monitoring video stream is a sequence of video frames captured by a camera mounted on the robot in real time and transmitted to a monitoring interface of the electronic device. Frame extraction can obtain multiple monitoring images.

[0036] Global motion estimation (GME) is used to determine a motion vector representing global motion, rather than local motion, between two adjacent monitoring images. For example, when the robot rotates or translates, the entire monitoring image captured by the camera will undergo global motion.

[0037] The candidate displacement composite search strategy is used to determine the search order of candidate displacements that need to be evaluated when performing global motion estimation based on motion spatiotemporal continuity, i.e., to preferentially search candidate displacements that are most likely to evaluate global motion, so that the cost function early stopping mechanism can be triggered earlier and more frequently, improving search efficiency and latency measurement efficiency.

[0038] The cost function early stopping mechanism is as follows: during the calculation of the cost function for each candidate displacement, since the cost function is non-negative, if the local cost cumulative sum of the candidate displacement is not less than the currently known minimum cost, the candidate displacement is a non-optimal displacement, and the calculation of the candidate displacement is terminated immediately, reducing the calculation amount of non-optimal displacements and saving a large number of CPU cycles, thereby improving the timeliness of latency measurement.

[0039] Specifically, after obtaining the trigger timestamp, the electronic device performs global motion estimation on the monitoring video stream captured by the robot through the cooperative work of the candidate displacement composite search strategy and the cost function early stopping mechanism, i.e., real-time judgment of whether the monitoring image undergoes global motion due to robot motion on the resource-limited electronic device, and when it is determined that the monitoring image undergoes global motion, the global timestamp of global motion is determined based on the system clock of the electronic device.

[0040] In step 130, the end-to-end latency of the robot is determined based on the global timestamp and the trigger timestamp.

[0041] Specifically, after determining the global timestamp and the trigger timestamp, a difference between the global timestamp and the trigger timestamp is calculated, and the difference is the end-to-end delay corresponding to the robot. Since the global timestamp and the trigger timestamp are from the system clock of the same electronic device, there is no problem of different device clock synchronization, the delay measurement error is avoided, and the accuracy of the delay measurement is ensured. Meanwhile, the trigger timestamp is taken as the starting point, and the global timestamp is taken as the end point, and the measurement link covers all nodes such as instruction issuing, robot parsing instruction, instruction execution, camera shooting, video stream uploading, and monitoring interface display, so that the end-to-end delay is the complete link delay from the electronic device to the electronic device.

[0042] The robot control end-to-end delay measurement method provided by the embodiment of the application comprises the following steps: after obtaining a trigger timestamp of a robot motion instruction, performing global motion estimation on a monitoring video stream of the robot according to a candidate displacement composite search strategy for adjusting a search order of candidate displacements between adjacent monitoring images based on motion space-time continuity and a cost function early termination mechanism for early termination of calculation of a cost function of the candidate displacement, obtaining a global timestamp of global motion of a monitoring image corresponding to the robot, and determining an end-to-end delay of the robot according to the global timestamp and the trigger timestamp. In the embodiment of the application, the end-to-end delay corresponding to the complete link from the user issuing the robot motion instruction on the client to the global motion of the monitoring image on the client is measured, and the trigger timestamp and the global timestamp are both determined based on the system clock of the same client, thereby avoiding the dynamic clock deviation caused by the different device system clock synchronization, improving the accuracy of the end-to-end delay measurement. Meanwhile, the candidate displacement composite search strategy and the cost function early termination mechanism are optimized cooperatively, the calculation amount of the global motion estimation is greatly reduced, the calculation efficiency of the delay measurement is improved, and the real-time performance of the end-to-end delay measurement is improved.

[0043] In one embodiment, the global motion estimation on the monitoring video stream corresponding to the robot based on the candidate displacement composite search strategy and the cost function early termination mechanism comprises the following steps: traversing the monitoring video stream corresponding to the robot to obtain a current frame monitoring image, a previous frame monitoring image and a previous frame best motion vector corresponding to the robot; determining a plurality of candidate displacements of the current frame based on the previous frame best motion vector and the candidate displacement composite search strategy; determining a current frame best motion vector and a global minimum cost based on the current frame monitoring image, the previous frame monitoring image, all candidate displacements and the cost function early termination mechanism; performing global motion estimation based on the current frame best motion vector and the global minimum cost to obtain a global motion estimation result corresponding to the current frame. In a case that the global motion estimation result represents that the robot corresponding monitoring image has global motion, stopping the iteration, and determining the time stamp corresponding to the current frame monitoring image as a global time stamp of the robot corresponding monitoring image having global motion.

[0044] Specifically, the monitoring video stream is frame extracted to obtain a monitoring image sequence, and the monitoring image sequence includes a plurality of monitoring images sorted in time sequence. All monitoring images are iterated in time sequence to obtain a current frame monitoring image, a previous frame monitoring image, and a previous frame best motion vector, the previous frame best motion vector being a displacement vector that minimizes the difference of the overlapping area between the previous frame monitoring image and the second previous frame monitoring image. Due to the spatiotemporal continuity of motion, the current frame best motion vector that minimizes the difference of the overlapping area between the current frame monitoring image and the previous frame monitoring image is most likely to appear near the previous frame best motion vector. Therefore, in the embodiment of the present application, the previous frame best motion vector is taken as a reference to search for a plurality of candidate displacements to be evaluated for the current frame according to a candidate displacement composite search strategy. Then, the cost function of each candidate displacement is calculated in combination with the current frame monitoring image, the previous frame monitoring image, and an early stopping mechanism of the cost function, so as to determine the final global minimum cost and the current frame best motion vector that generates the global minimum cost. In the calculation process, if the early stopping mechanism of the cost function is triggered, the subsequent calculation of the cost function of the candidate displacement can be terminated, and the calculation of the cost function of the next candidate displacement is continued, so as to reduce the calculation amount of non-optimal candidate displacement, thereby improving the calculation efficiency of determining the current frame best motion vector. The global minimum cost is used to represent that the difference of the overlapping area of the current frame monitoring image and the previous frame monitoring image is minimized after motion compensation by the current frame best motion vector. After the current frame best motion vector and the global minimum cost are determined, global motion estimation is performed according to the current frame best motion vector and the global minimum cost to obtain a global motion estimation result. The global motion estimation result includes that the monitoring image has global motion or the monitoring image has no obvious motion. In a case that the global motion estimation result is that the monitoring image has global motion, a time stamp attached to the current frame monitoring image is determined as a global time stamp.

[0045] In addition, in a case that the determined global motion estimation result is that the monitoring image has no obvious motion, the current frame monitoring image can be taken as a new previous frame monitoring image, the next frame monitoring image can be taken as a new current frame monitoring image, and the above steps can be repeatedly executed until it is determined that the global motion estimation result is that the monitoring image has global motion, and then the global time stamp is determined.

[0046] In one embodiment, the candidate displacement composite search strategy at least includes a neighborhood priority search strategy, an axial constraint search strategy, and an extended ring iteration search strategy. The neighborhood priority search strategy is used to search candidate displacements of the current frame in order of distance from the previous frame best motion vector from small to large; The axial constraint search strategy is used to constrain the search step size of the current frame in a second motion direction based on a first motion direction of the previous frame; the first motion direction is perpendicular to the second motion direction; The extended ring traversal search strategy is used to search candidate displacements of the current frame in order of Manhattan distance from the origin from small to large for a remaining displacement; the remaining displacement is a displacement remaining in the search space after the neighborhood priority search strategy is performed.

[0047] Specifically, in order to achieve extremely high calculation efficiency on the premise of ensuring measurement accuracy, the traditional global traversal search is not used in the embodiments of the present application, but a candidate displacement composite search strategy is used to reduce the number of candidate displacements that need to be calculated, so that better candidate displacements are calculated first, so that the cost function early stopping mechanism can be triggered earlier and more frequently, and the calculation efficiency is maximized. The candidate displacement composite search strategy at least includes a neighborhood priority search strategy, an axial constraint search strategy and an extended ring traversal search strategy.

[0048] The neighborhood priority search strategy is: due to the spatiotemporal continuity of motion, the current frame best motion vector has a great probability of appearing in a small range neighborhood centered on the previous frame best motion vector, therefore, the candidate displacements in the small range neighborhood centered on the previous frame best motion vector are preferentially checked, that is, the candidate displacements close to the previous frame best motion vector are preferentially checked, so that the threshold of the cost function early stopping mechanism can be quickly reduced to a lower level. The previous frame best motion vector is (prevBestDx, prevBestDy), prevBestDx represents the previous frame best horizontal direction displacement, and prevBestDy represents the previous frame best vertical direction displacement.

[0049] The axial constraint search strategy is: pruning according to the main motion direction (i.e. the first motion direction) of the previous frame, constraining the search step size of the second motion direction, effectively excluding large displacements perpendicular to the first motion direction, and significantly reducing the search space. For example, if the absolute value of prevBestDx is greater than the absolute value of prevBestDy in the previous frame best motion vector, i.e. abs(prevBestDx)>abs(prevBestDy), it indicates that the first motion direction of the previous frame is the horizontal direction. Therefore, in the search process of the current frame, the current frame vertical displacement (dy) in the second motion direction (i.e. the vertical direction) is limited, for example, the absolute value of the current frame vertical displacement is constrained to be less than or equal to 2, i.e. abs(dy)≤2.

[0050] The extended ring traversal search strategy is: after performing the neighborhood priority search strategy, for the remaining displacements in the search space, the Manhattan distance between each remaining displacement and the origin (0, 0) is calculated, and the ring traversal is performed in the order of the Manhattan distance from small to large. For example, the remaining displacements with a Manhattan distance of 1 ((1, 0), (-1, 0), (0, 1) and (0, -1)) are traversed first, and then the remaining displacements with a Manhattan distance of 2 are traversed. In this way, the traversal is performed until the maximum displacement is reached. The order of the ring traversal ensures that the smaller remaining displacements are evaluated first, thereby enhancing the effect of the early termination mechanism of the cost function.

[0051] It should be noted that the Manhattan distance is the sum of the absolute value of the horizontal displacement (dx) of the current frame and the absolute value of the vertical displacement (dy) of the current frame, that is, |dx|+|dy|.

[0052] In addition, the candidate displacement composite search strategy also includes a history optimal reset search strategy, which is used to reset the best motion vector of the previous frame. That is, if the cost reduction rate calculated for the current frame is very low, for example, the cost reduction rate is less than 5%, indicating that the motion signal is weak or unreliable, at this time, the best motion vector of the previous frame can be reset to (0, 0) to prevent the accumulation of false noise from affecting the neighborhood search of the next frame.

[0053] Optionally, the candidate displacement composite search strategy can also include a three-step search strategy, a diamond search strategy, etc., which is not limited by the embodiments of the present application.

[0054] In one embodiment, the determination of the current frame best motion vector and the global minimum cost based on the current frame monitoring image, the previous frame monitoring image, all candidate displacements and the early termination mechanism of the cost function comprises: S1, determining a current frame gray image corresponding to the current frame monitoring image and a previous frame gray image corresponding to the previous frame monitoring image; S2, traversing all candidate displacements, and determining a final cost cumulative sum of the current candidate displacement that has not triggered the early termination mechanism of the cost function based on the current frame gray image, the previous frame gray image and the current candidate displacement; S3, determining the final cost cumulative sum as a new current minimum cost, and determining a next candidate displacement as a new current candidate displacement; S4, repeatedly performing step S2 and step S3 until all candidate displacements are traversed, and determining a final current minimum cost as a final minimum cost; S5, determining the current frame best motion vector and the global minimum cost based on the final minimum cost.

[0055] Specifically, after obtaining the current frame monitoring image and the previous frame monitoring image, the current frame monitoring image and the previous frame monitoring image are respectively grayed to obtain a current frame gray image corresponding to the current frame monitoring image and a previous frame gray image corresponding to the previous frame monitoring image, so as to reduce the number of pixels participating in subsequent calculation. Then, all candidate displacements are traversed, and a cost function is calculated in combination with the current frame gray image, the previous frame gray image and the current candidate displacement, that is, the current frame gray image and the previous frame gray image are motion compensated through the current candidate displacement, and it is judged whether the cost function early stop mechanism is triggered in the calculation process. If the cost function early stop mechanism is triggered, the subsequent calculation of the current candidate displacement is terminated, and the cost function calculation of the next candidate displacement is continued. If the cost function early stop mechanism is not triggered in the calculation process, a final cost cumulative sum of the current candidate displacement can be obtained, and the final cost cumulative sum is less than the current minimum cost. Then, the final cost cumulative sum can be determined as a new current minimum cost, and the cost function calculation of the next candidate displacement is continued. The above operation is repeated until all candidate displacements are traversed, and the final current minimum cost can be determined as a final minimum cost. According to the final minimum cost, the current frame best motion vector and the global minimum cost can be determined.

[0056] In an embodiment, the current frame gray image corresponding to the current frame monitoring image and the previous frame gray image corresponding to the previous frame monitoring image are determined by:

[0057] Specifically, before obtaining the current frame monitoring image and the previous frame monitoring image, and when the electronic device player is initialized or the video resolution is changed, the preset down-sampling size is set in advance, for example, the preset down-sampling size is 160*90, and the maximum displacement of the search window is determined, for example, the maximum displacement (maxShift) is 16 pixels. In order to reduce the calculation overhead, the pre-processing interval of the monitoring image can also be set, for example, the pre-processing interval is 3 frames. That is, after obtaining the previous frame monitoring image, the current frame monitoring image is obtained after an interval of 3 frames.

[0058] After obtaining the current frame monitoring image and the previous frame monitoring image, the current frame monitoring image and the previous frame monitoring image are both of the original resolution 1920*1080. Then, Figure 2 is a schematic diagram of the current frame monitoring image before and after down-sampling provided by the embodiment of the application, as​​Figure 2 As shown in FIG. 1, the current frame monitoring image is efficiently down-sampled from the original resolution to a preset down-sampling size by a frame buffer object (FBO) rendering technology of a graphics processing unit (GPU) to obtain a current frame down-sampled image. Figure 3 FIG. 2 is a schematic diagram of the previous frame monitoring image before and after down-sampling provided by an embodiment of the present application, as shown in FIG. 2, the previous frame monitoring image is efficiently down-sampled from the original resolution to a preset down-sampling size to obtain a previous frame down-sampled image. Figure 3 As shown in FIG. 2, similarly, the previous frame monitoring image is efficiently down-sampled from the original resolution to a preset down-sampling size to obtain a previous frame down-sampled image. After down-sampling, the resolutions of the current frame down-sampled image and the previous frame down-sampled image are both 160x90, and the image data formats of the current frame down-sampled image and the previous frame down-sampled image are both RGBA (Red Green Blue Alpha) formats.

[0059] After that, in order to simplify the calculation of the subsequent cost function, the current frame down-sampled image and the previous frame down-sampled image are both grayscaled by using a luminance (Luma) formula defined in the BT.601 standard to obtain a current frame grayscale image and a previous frame grayscale image. The luminance formula is Y=(77xR+150xG+29xB)>>8. The BT.601 standard is a color encoding standard defined in the television broadcast standard of the International Telecommunication Union. According to the luminance formula, the luminance value is calculated by weighted average of each pixel in the current frame down-sampled image and the previous frame down-sampled image according to the sensitivity difference of the human eye to light of different wavelengths, and the luminance value is taken as the grayscale value, so that the color image is converted into the grayscale image. In the luminance formula, 77 represents the weight of the sensitivity of the human eye to the red component, 150 represents the weight of the sensitivity of the human eye to the green component, 29 represents the weight of the sensitivity of the human eye to the blue component, >> represents the right shift operator, and >>8 represents the right shift of 8 bits, which is equivalent to dividing by 2 8 .

[0060] In the embodiment, by down-sampling the current frame monitoring image and the previous frame monitoring image, the number of pixels participating in subsequent calculation is greatly reduced, the grayscale further compresses the three-channel data into single-channel data, the memory occupation and the subsequent calculation amount are greatly reduced, and the mobile device such as a mobile phone can also run in real time with extremely low CPU occupation, without affecting the user experience.

[0061] In one embodiment, the determining, based on the current frame grayscale image, the previous frame grayscale image and the current candidate displacement, of the final cost cumulative sum of the current candidate displacement that does not trigger the early stopping mechanism of the cost function at all, comprises: S21, determining, based on the current candidate displacement, an overlapping area of the current frame grayscale image and the previous frame grayscale image; S22, traverse each pixel in the overlapping region, determine a local cost accumulation corresponding to the current candidate displacement based on a current pixel value of a target pixel in the current frame grayscale image, a previous pixel value of the target pixel in the previous frame grayscale image, and the current candidate displacement, the target pixel including the current pixel and all historical pixels before the current pixel; S23, in a case where the local cost accumulation is greater than or equal to the current minimum cost, trigger the cost function early stopping mechanism to terminate the calculation of the current candidate displacement, determine the next candidate displacement as a new current candidate displacement, and repeat steps S2 and S5; S24, in a case where the local cost accumulation is less than the current minimum cost, add a next pixel to the target pixel, and repeat steps S22 and S24 based on the new target pixel until all pixels in the overlapping region are traversed, and determine a final local cost accumulation as the final cost accumulation of the current candidate displacement without triggering the cost function early stopping mechanism all the time.

[0062] The local cost accumulation is calculated based on a cost function, which can be a sum of absolute differences (SAD), a sum of squared differences (SSD), or a normalized cross-correlation (NCC) coefficient, and the following description is based on the cost function being a sum of absolute differences.

[0063] The overlapping region is a pixel region corresponding to the current frame grayscale image and the compensated previous frame grayscale image after the previous frame grayscale image is motion compensated by the candidate displacement, and the size of the overlapping region depends on the candidate displacement and the image sizes of the current frame grayscale image and the previous frame grayscale image. The total number of pixels in the overlapping region is different for different candidate displacements.

[0064] Specifically, after the overlapping region between the current frame grayscale image and the previous frame grayscale image is determined according to the current candidate displacement, all pixels in the overlapping region are traversed, and formula (1) is used to calculate the local cost accumulation corresponding to the target pixel, that is, the absolute difference values of the first pixel to the current pixel under the current candidate displacement are calculated, the absolute difference values corresponding to the first pixel to the current pixel are summed, and the sum is the local cost accumulation of the target pixel. Formula (1) is: .

[0065] wherein dx represents a current frame horizontal displacement of the current candidate displacement; dy represents a current frame vertical displacement of the current candidate displacement; k represents a kth pixel in the overlap region, i.e. the current pixel, the target pixels include the 1st to the kth pixels in the overlap region, and k is an integer greater than or equal to 1 and less than or equal to the total number of pixels in the overlap region; denotes a local cost summation of the target pixels; denotes a pixel coordinate of the ith pixel in the overlap region, and 1≤i≤k, denotes a current pixel value at the pixel coordinate of the ith pixel in the current frame grayscale image, denotes a previous pixel value at the pixel coordinate of the ith pixel in the previous frame grayscale image after displacement compensation, denotes an absolute difference value of the ith pixel under the current candidate displacement.

[0066] Then, the local cost summation is compared with the current minimum cost. Since the absolute difference value is non-negative, the sum of the absolute difference values of the k pixels is monotonically non-decreasing, i.e. the local cost summation is monotonically non-decreasing, i.e. If the local cost summation of the k pixels is greater than or equal to the current minimum cost, i.e. then the final cost summation of all pixels in the overlap region under the current candidate displacement is also necessarily greater than or equal to the current minimum cost, which means that the final cost summation corresponding to the current candidate displacement cannot be better than the current minimum cost, so the subsequent calculation of the current candidate displacement can be safely terminated in advance. Then, the next candidate displacement is taken as a new current candidate displacement, and the above operation is repeated. If the local cost summation of the k pixels is less than the current minimum cost, then the final cost summation of all pixels in the overlap region under the current candidate displacement can be less than the current minimum cost, in which case the absolute difference value of the (k+1)th pixel under the current candidate displacement is calculated, so as to calculate a new local cost summation of the new target pixels including the 1st to the (k+1)th pixels, and the above judgment and summation operation is repeated. If the early termination mechanism of the cost function is not triggered after traversing all pixels in the overlap region, it indicates that the final cost summation of the current candidate displacement is less than the current minimum cost, in which case the value of the current minimum cost is replaced by the value of the final cost summation of the current candidate displacement to dynamically maintain the current minimum cost, so as to ensure that the current minimum cost is the minimum. Then, the next candidate displacement is taken as a new current candidate displacement, and the above operation is repeated until all candidate displacements are traversed, and the final current minimum cost is determined as the final minimum cost, so as to determine the current frame best motion vector and the global minimum cost subsequently.

[0067] In the embodiment, the cost function early stopping mechanism is used to make the cost function of most non-optimal candidate displacements only partially calculated, avoid the waste of computing resources and time caused by the complete calculation of non-optimal candidate displacements, save a large number of CPU cycles, and thus improve the efficiency of time delay measurement.

[0068] In one embodiment, the determining the global minimum cost based on the final minimum cost comprises: determining the global minimum cost based on the final minimum cost and the total number of pixels in the overlapping region; determining the candidate displacement corresponding to the final minimum cost as the optimal motion vector of the current frame.

[0069] Specifically, after the final minimum cost is determined, the mean value (i.e., the average absolute difference) of the final minimum cost is calculated according to the total number of pixels in the overlapping region by using formula (2), and the mean value is the global minimum cost. Formula (2) is: .

[0070] wherein, the mean value is the global minimum cost, and N represents the total number of pixels in the overlapping region.

[0071] Meanwhile, the candidate displacement corresponding to the final minimum cost can be determined, and the candidate displacement is the optimal motion vector of the current frame, i.e., the optimal motion vector of the current frame can make the difference between the overlapping region of the current frame and the previous frame gray scale image minimum.

[0072] In one embodiment, the global motion estimation based on the optimal motion vector of the current frame and the global minimum cost comprises: determining a cost reduction rate based on the non-shift global cost and the global minimum cost; determining the global motion estimation result corresponding to the current frame based on the non-shift global cost, the cost reduction rate, the global minimum cost, and the length of the module corresponding to the optimal motion vector of the current frame.

[0073] Specifically, after the optimal motion vector of the current frame and the global minimum cost are determined, dx=0 and dy=0 are substituted into formula (2) to calculate the non-shift global cost, which can be represented as madNoShift. The non-shift global cost is used to measure the motion significance of the monitoring image. If the non-shift global cost is less than a noise threshold, it indicates that the monitoring image has no obvious motion, and the robot is in a static scene or a low-noise scene. If the non-shift global cost is greater than or equal to the noise threshold, it indicates that the monitoring image has obvious motion, and the robot is in a dynamic scene.

[0074] Then, a cost reduction ratio is calculated according to the non-shift global cost and the global minimum cost by using formula (3), and the cost reduction ratio is used to measure the significance and reliability of the motion signal, that is, the closer the cost reduction ratio is to 1, the stronger the explanatory ability of the determined current frame best motion vector to the change of the monitoring image, the more significant and reliable the motion signal is, and vice versa, the closer the cost reduction ratio is to 1, the weaker and less reliable the motion signal is. Formula (3) is: reductionRatio = (madNoShift - madBest) / madNoShift.

[0075] Wherein, reductionRatio represents the cost reduction ratio, and madBest represents the global minimum cost.

[0076] Then, in combination with the non-shift global cost, the cost reduction ratio, the global minimum cost and the length of the current frame best motion vector, it is determined whether the monitoring image of the robot has global motion, and a global motion estimation result is obtained.

[0077] Optionally, the noise threshold can be 4.0, 4.1 or 4.2, and the embodiments of the present application do not limit this.

[0078] In one embodiment, the global motion estimation result is determined to be that the monitoring image corresponding to the robot has global motion, comprising: In the case that the non-shift global cost is greater than or equal to the noise threshold, the cost reduction ratio is greater than a first preset threshold, the global minimum cost is greater than 0, and the length of the current frame best motion vector is greater than a second preset threshold, the global motion estimation result is determined to be that the monitoring image corresponding to the robot has global motion.

[0079] Specifically, after obtaining the global motion estimation result, if the non-shift global cost is greater than or equal to the noise threshold, it indicates that there is obvious motion in the monitoring image, if the cost reduction ratio is greater than the first preset threshold, it indicates that the strength and reliability of the motion signal are high, if the global minimum cost is greater than 0, it indicates that the content in the monitoring image is in motion, rather than the translation of a still picture, and if the length of the current frame best motion vector is greater than the second preset threshold, it indicates that the motion amplitude of the monitoring image is large, and according to the sign and displacement size of the current frame best motion vector, the motion direction can be further determined, for example, left translation, right translation, upward tilt, downward tilt, etc. In the case that all the above conditions are met, it can be determined that the monitoring image corresponding to the robot has global motion, that is, the global motion estimation result is that the monitoring image corresponding to the robot has global motion.

[0080] It should be noted that the noise threshold, the first preset threshold and the second preset threshold can be set according to historical experience, and the embodiments of the present application do not limit this.

[0081] It should be noted that the embodiments of the present application are applied to electronic equipment (i.e. client), without any firmware or hardware modification to the robot, suitable for robots of different manufacturers and different models, only need to provide standard monitoring video stream, has strong universality and compatibility. And due to the substantial reduction of the amount of calculation, the calculation efficiency is greatly improved, the time delay measurement of single motion is extremely short, usually in millisecond level, can make nearly instantaneous response to the motion of monitoring picture, thereby ensuring the real-time of time delay measurement and the accuracy of time delay measurement result.

[0082] The robot control end-to-end time delay measurement device provided by the present application is described below, and the robot control end-to-end time delay measurement device described below can be mutually corresponding to the robot control end-to-end time delay measurement method described above.

[0083] The robot control end-to-end time delay measurement device provided by the present application is described below, and the robot control end-to-end time delay measurement device described below can be mutually corresponding to the robot control end-to-end time delay measurement method described above. Figure 4 Figure 4 As shown in the structure diagram of the robot control end-to-end time delay measurement device provided by the present application, the robot control end-to-end time delay measurement device 400 comprises an acquisition module 410, a global motion estimation module 420 and a determination module 430.

[0084] The acquisition module 410 is configured to acquire a trigger timestamp of a robot motion instruction.

[0085] The global motion estimation module 420 is configured to perform global motion estimation on a monitoring video stream corresponding to the robot based on a candidate displacement composite search strategy and a cost function early stopping mechanism, to obtain a global timestamp of global motion of a monitoring image in the monitoring video stream; the candidate displacement composite search strategy is configured to adjust a search order of candidate displacements between adjacent monitoring images based on motion spatiotemporal continuity; in a case that a candidate displacement triggers the cost function early stopping mechanism, the cost function early stopping mechanism is configured to terminate calculation of a cost function of the candidate displacement in advance.

[0086] The determination module 430 is configured to determine an end-to-end time delay corresponding to the robot based on the global timestamp and the trigger timestamp.

[0087] ​The robot control end-to-end delay measurement device provided in this application, after acquiring the trigger timestamp of the robot motion command, performs global motion estimation on the robot's monitoring video stream based on a candidate displacement composite search strategy for adjusting the search order of candidate displacements between adjacent monitoring images based on the spatiotemporal continuity of motion, and a cost function early stopping mechanism for prematurely terminating the calculation of the cost function for candidate displacements. This yields the global timestamp of the global motion occurring in the monitoring image corresponding to the robot. Based on the global timestamp and the trigger timestamp, the robot's end-to-end delay is determined. In this application embodiment, the measurement is of the end-to-end delay corresponding to the complete link from the user issuing the robot motion command to the monitoring image on the client, where global motion occurs. Both the trigger timestamp and the global timestamp are determined based on the system clock of the same client, avoiding dynamic clock deviations caused by asynchronous system clocks of different devices, thus improving the accuracy of end-to-end delay measurement. Simultaneously, through the synergistic optimization of the candidate displacement composite search strategy and the cost function early stopping mechanism, the computational load of global motion estimation is significantly reduced, improving the computational efficiency of delay measurement, thereby enhancing the real-time performance of end-to-end delay measurement.

[0088] Optionally, the global motion estimation module 420 is specifically used for: Traverse the monitoring video stream corresponding to the robot to obtain the current frame monitoring image, the previous frame monitoring image, and the previous frame best motion vector corresponding to the robot; Based on the combined search strategy of the best motion vector of the previous frame and the candidate displacement, multiple candidate displacements of the current frame are determined. Based on the current frame monitoring image, the previous frame monitoring image, all candidate displacements, and the cost function early stopping mechanism, determine the optimal motion vector and the global minimum cost for the current frame; Global motion estimation is performed based on the current frame's optimal motion vector and the global minimum cost to obtain the global motion estimation result for the current frame. If the global motion estimation result indicates that the monitoring image corresponding to the robot has undergone global motion, the traversal is stopped, and the timestamp corresponding to the current frame is determined as the global timestamp of the monitoring image corresponding to the robot undergoing global motion.

[0089] Optionally, the candidate displacement composite search strategy includes at least a neighborhood-first search strategy, an axial constraint search strategy, and an extended cyclic traversal search strategy; The neighborhood-first search strategy is used to search for candidate displacements in the current frame in ascending order of distance from the best motion vector of the previous frame. The axial constraint search strategy is used to constrain the search step size of the current frame in the second motion direction based on the first motion direction of the previous frame; the first motion direction is perpendicular to the second motion direction. The extended ring traversal search strategy is used to search for candidate displacements of the current frame in ascending order of Manhattan distance from the origin for the remaining displacements; the remaining displacements are the displacements remaining in the search space after the neighborhood-first search strategy is executed.

[0090] Optionally, the global motion estimation module 420 is specifically used for: S1. Determine the grayscale image of the current frame corresponding to the current frame monitoring image and the grayscale image of the previous frame corresponding to the previous frame monitoring image. S2. Traverse all candidate displacements and, based on the current frame grayscale image, the previous frame grayscale image, and the current candidate displacement, determine the final cost sum of the current candidate displacements that have never triggered the early stopping mechanism of the cost function. S3. The sum of the final costs is determined as the new current minimum cost, and the next candidate displacement is determined as the new current candidate displacement; S4. Repeat steps S2 and S3 until all candidate displacements have been traversed, and determine the current minimum cost as the final minimum cost. S5. Based on the final minimum cost, determine the optimal motion vector for the current frame and the global minimum cost.

[0091] Optionally, the global motion estimation module 420 is specifically used for: S21. Based on the current candidate displacement, determine the overlapping area between the current frame grayscale image and the previous frame grayscale image; S22. Traverse each pixel in the overlapping region, and based on the current pixel value of the target pixel in the current frame grayscale image, the previous pixel value of the target pixel in the previous frame grayscale image, and the current candidate displacement, determine the local cost accumulation sum corresponding to the current candidate displacement; the target pixel includes the current pixel and all historical pixels before the current pixel. S23. If the sum of the local costs is greater than or equal to the current minimum cost, trigger the early stopping mechanism of the cost function to terminate the calculation of the current candidate displacement; determine the next candidate displacement as the new current candidate displacement, and repeat steps S2 and S5. S24. If the sum of local costs is less than the current minimum cost, add the next pixel to the target pixel, and repeat steps S22 and S24 based on the new target pixel until all pixels in the overlapping region have been traversed. The final sum of local costs is determined as the final sum of costs for which the current candidate displacement has never triggered the early stopping mechanism of the cost function.

[0092] Optionally, the global motion estimation module 420 is specifically used for: The global minimum cost is determined based on the final minimum cost and the total number of pixels in the overlapping region; The candidate displacement corresponding to the final minimum cost is determined as the optimal motion vector for the current frame.

[0093] Optionally, the global motion estimation module 420 is specifically used for: Based on a preset downsampling size, the current frame monitoring image and the previous frame monitoring image are downsampled respectively to obtain the current frame downsampled image corresponding to the current frame monitoring image and the previous frame downsampled image corresponding to the previous frame monitoring image; The current frame downsampled image and the previous frame downsampled image are converted to grayscale to obtain the current frame grayscale image corresponding to the current frame downsampled image and the previous frame grayscale image corresponding to the previous frame downsampled image.

[0094] Optionally, the global motion estimation module 420 is specifically used for: The cost reduction rate is determined based on the global cost without displacement and the global minimum cost. Based on the displacement-free global cost, the cost reduction rate, the global minimum cost, and the magnitude corresponding to the best motion vector of the current frame, the global motion estimation result corresponding to the current frame is determined.

[0095] Optionally, the global motion estimation module 420 is specifically used for: If the global cost without displacement is greater than or equal to the noise threshold, the cost reduction rate is greater than the first preset threshold, the global minimum cost is greater than 0, and the magnitude corresponding to the best motion vector in the current frame is greater than the second preset threshold, then the global motion estimation result is determined to be that the monitoring image corresponding to the robot has undergone global motion.

[0096] Figure 5 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application, such as... Figure 5As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute a robot control end-to-end delay measurement method. This method includes: obtaining the trigger timestamp of the robot motion command; performing global motion estimation on the monitoring video stream corresponding to the robot based on a candidate displacement composite search strategy and a cost function early stopping mechanism to obtain a global timestamp of the global motion of the monitoring image in the monitoring video stream; the candidate displacement composite search strategy is used to adjust the search order of candidate displacements between adjacent monitoring images based on the spatiotemporal continuity of motion; when the candidate displacement triggers the cost function early stopping mechanism, the cost function early stopping mechanism is used to terminate the calculation of the cost function of the candidate displacement in advance; and determining the end-to-end delay corresponding to the robot based on the global timestamp and the trigger timestamp.

[0097] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0098] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the robot control end-to-end delay measurement method provided by the above methods. The method includes: obtaining the trigger timestamp of the robot motion command; performing global motion estimation on the monitoring video stream corresponding to the robot based on a candidate displacement composite search strategy and a cost function early stopping mechanism to obtain a global timestamp of the global motion of the monitoring image in the monitoring video stream; the candidate displacement composite search strategy is used to adjust the search order of candidate displacements between adjacent monitoring images based on the spatiotemporal continuity of motion; when the candidate displacement triggers the cost function early stopping mechanism, the cost function early stopping mechanism is used to terminate the calculation of the cost function of the candidate displacement in advance; and determining the end-to-end delay corresponding to the robot based on the global timestamp and the trigger timestamp.

[0099] In another aspect, this application also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the robot control end-to-end delay measurement method provided by the above methods. The method includes: obtaining the trigger timestamp of the robot motion command; performing global motion estimation on the monitoring video stream corresponding to the robot based on a candidate displacement composite search strategy and a cost function early stopping mechanism to obtain a global timestamp of the global motion of the monitoring image in the monitoring video stream; the candidate displacement composite search strategy is used to adjust the search order of candidate displacements between adjacent monitoring images based on the spatiotemporal continuity of motion; when the candidate displacement triggers the cost function early stopping mechanism, the cost function early stopping mechanism is used to terminate the calculation of the cost function of the candidate displacement in advance; and determining the end-to-end delay corresponding to the robot based on the global timestamp and the trigger timestamp.

[0100] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0101] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0102] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for measuring end-to-end time delay in robot control, characterized in that, include: Obtain the trigger timestamp of the robot's motion command; Based on a candidate displacement composite search strategy and a cost function early stopping mechanism, global motion estimation is performed on the monitoring video stream corresponding to the robot to obtain the global timestamp of the global motion of the monitoring images in the monitoring video stream; the candidate displacement composite search strategy is used to adjust the search order of candidate displacements between adjacent monitoring images based on the spatiotemporal continuity of motion; when a candidate displacement triggers the cost function early stopping mechanism, the cost function early stopping mechanism is used to terminate the calculation of the cost function of the candidate displacement in advance. Based on the global timestamp and the trigger timestamp, the end-to-end latency corresponding to the robot is determined.

2. The robot control end-to-end time delay measurement method according to claim 1, characterized in that, The method based on the candidate displacement composite search strategy and the cost function early stopping mechanism performs global motion estimation on the monitoring video stream corresponding to the robot, and obtains the global timestamps of global motion occurrences in the monitoring images in the monitoring video stream, including: Traverse the monitoring video stream corresponding to the robot to obtain the current frame monitoring image, the previous frame monitoring image, and the previous frame best motion vector corresponding to the robot; Based on the combined search strategy of the best motion vector of the previous frame and the candidate displacement, multiple candidate displacements of the current frame are determined. Based on the current frame monitoring image, the previous frame monitoring image, all candidate displacements, and the cost function early stopping mechanism, determine the optimal motion vector and the global minimum cost for the current frame; Global motion estimation is performed based on the current frame's optimal motion vector and the global minimum cost to obtain the global motion estimation result for the current frame. If the global motion estimation result indicates that the monitoring image corresponding to the robot has undergone global motion, the traversal is stopped, and the timestamp corresponding to the current frame is determined as the global timestamp of the monitoring image corresponding to the robot undergoing global motion.

3. The robot control end-to-end time delay measurement method according to claim 2, characterized in that, The candidate displacement composite search strategy includes at least a neighborhood-first search strategy, an axial constraint search strategy, and an extended circumferential search strategy. The neighborhood-first search strategy is used to search for candidate displacements in the current frame in ascending order of distance from the best motion vector of the previous frame. The axial constraint search strategy is used to constrain the search step size of the current frame in the second motion direction based on the first motion direction of the previous frame; the first motion direction is perpendicular to the second motion direction. The extended ring traversal search strategy is used to search for candidate displacements of the current frame in ascending order of Manhattan distance from the origin for the remaining displacements; the remaining displacements are the displacements remaining in the search space after the neighborhood-first search strategy is executed.

4. The robot control end-to-end time delay measurement method according to claim 2, characterized in that, The process of determining the optimal motion vector and global minimum cost for the current frame based on the current frame monitoring image, the previous frame monitoring image, all candidate displacements, and the cost function early stopping mechanism includes: S1. Determine the grayscale image of the current frame corresponding to the current frame monitoring image and the grayscale image of the previous frame corresponding to the previous frame monitoring image. S2. Traverse all candidate displacements and, based on the current frame grayscale image, the previous frame grayscale image, and the current candidate displacement, determine the final cost sum of the current candidate displacements that have never triggered the early stopping mechanism of the cost function. S3. The sum of the final costs is determined as the new current minimum cost, and the next candidate displacement is determined as the new current candidate displacement; S4. Repeat steps S2 and S3 until all candidate displacements have been traversed, and determine the current minimum cost as the final minimum cost. S5. Based on the final minimum cost, determine the optimal motion vector for the current frame and the global minimum cost.

5. The robot control end-to-end time delay measurement method according to claim 4, characterized in that, The step of determining the final cost summation based on the current frame grayscale image, the previous frame grayscale image, and the current candidate displacement, where the current candidate displacement has never triggered the early stopping mechanism of the cost function, includes: S21. Based on the current candidate displacement, determine the overlapping area between the current frame grayscale image and the previous frame grayscale image; S22. Traverse each pixel in the overlapping region, and based on the current pixel value of the target pixel in the current frame grayscale image, the previous pixel value of the target pixel in the previous frame grayscale image, and the current candidate displacement, determine the local cost accumulation sum corresponding to the current candidate displacement; the target pixel includes the current pixel and all historical pixels before the current pixel. S23. If the sum of the local costs is greater than or equal to the current minimum cost, trigger the early stopping mechanism of the cost function to terminate the calculation of the current candidate displacement; determine the next candidate displacement as the new current candidate displacement, and repeat steps S2 and S5. S24. If the sum of local costs is less than the current minimum cost, add the next pixel to the target pixel, and repeat steps S22 and S24 based on the new target pixel until all pixels in the overlapping region have been traversed. The final sum of local costs is determined as the final sum of costs for which the current candidate displacement has never triggered the early stopping mechanism of the cost function.

6. The robot control end-to-end time delay measurement method according to claim 5, characterized in that, The process of determining the optimal motion vector for the current frame and the global minimum cost based on the final minimum cost includes: The global minimum cost is determined based on the final minimum cost and the total number of pixels in the overlapping region; The candidate displacement corresponding to the final minimum cost is determined as the optimal motion vector for the current frame.

7. The robot control end-to-end time delay measurement method according to claim 4, characterized in that, Determining the grayscale image of the current frame corresponding to the current frame monitoring image and the grayscale image of the previous frame corresponding to the previous frame monitoring image includes: Based on a preset downsampling size, the current frame monitoring image and the previous frame monitoring image are downsampled respectively to obtain the current frame downsampled image corresponding to the current frame monitoring image and the previous frame downsampled image corresponding to the previous frame monitoring image; The current frame downsampled image and the previous frame downsampled image are converted to grayscale to obtain the current frame grayscale image corresponding to the current frame downsampled image and the previous frame grayscale image corresponding to the previous frame downsampled image.

8. The robot control end-to-end time delay measurement method according to claim 2, characterized in that, The step of performing global motion estimation based on the best motion vector of the current frame and the global minimum cost to obtain the global motion estimation result corresponding to the current frame includes: The cost reduction rate is determined based on the global cost without displacement and the global minimum cost. Based on the displacement-free global cost, the cost reduction rate, the global minimum cost, and the magnitude corresponding to the best motion vector of the current frame, the global motion estimation result corresponding to the current frame is determined.

9. The robot control end-to-end time delay measurement method according to claim 8, characterized in that, Determining that the global motion estimation result indicates that the monitoring image corresponding to the robot has undergone global motion includes: If the global cost without displacement is greater than or equal to the noise threshold, the cost reduction rate is greater than the first preset threshold, the global minimum cost is greater than 0, and the magnitude corresponding to the best motion vector in the current frame is greater than the second preset threshold, then the global motion estimation result is determined to be that the monitoring image corresponding to the robot has undergone global motion.

10. A robot control end-to-end time delay measurement device, characterized in that, include: The acquisition module is used to acquire the trigger timestamp of the robot's motion commands; The global motion estimation module is used to perform global motion estimation on the monitoring video stream corresponding to the robot based on a candidate displacement composite search strategy and a cost function early stopping mechanism, and obtain the global timestamp of the global motion of the monitoring images in the monitoring video stream; the candidate displacement composite search strategy is used to adjust the search order of candidate displacements between adjacent monitoring images based on the spatiotemporal continuity of motion; when a candidate displacement triggers the cost function early stopping mechanism, the cost function early stopping mechanism is used to terminate the calculation of the cost function of the candidate displacement in advance; The determination module is used to determine the end-to-end latency corresponding to the robot based on the global timestamp and the trigger timestamp.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the robot control end-to-end delay measurement method as described in any one of claims 1 to 9.

12. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the robot control end-to-end delay measurement method as described in any one of claims 1 to 9.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the robot control end-to-end delay measurement method as described in any one of claims 1 to 9.

Images