Time synchronization judgment method for image acquisition of automatic operation equipment and cleaning robot

Abstract

The invention relates to a time synchronization judgment method for image acquisition of automatic operation equipment and a cleaning robot, and belongs to the technical field of image acquisition, and the method comprises the steps that a first processing unit sends a current trigger signal used for triggering image acquisition to a structured light camera, and generates a first counting identifier and a first timestamp of the current trigger signal; the second processing unit obtains the current image and generates a second counting identifier of the current image; based on the first counting identifier and the second counting identifier, judging whether the current image acquisition time is synchronous with the current trigger signal generation time or not, and if the current image acquisition time is synchronous with the current trigger signal generation time but the second counting identifier is inconsistent with the first counting identifier, according to the second counting identifier of the current image and a plurality of historical trigger signals; and taking the first timestamp of the matched historical trigger signal as the timestamp of the current image. According to the scheme, the timestamps can be accurately allocated to the images when the counting identifiers are not matched due to transmission delay and the like, and then the control precision of the automatic operation equipment is improved.

Description

[0001] This application is a divisional application of Chinese invention patent application filed by the applicant on October 13, 2022, entitled "Method for Time Synchronization Judgment of Image Acquisition of Automatic Operation Equipment and Cleaning Robot", application number 202211253496.4. Technical Field

[0002] This application belongs to the field of image acquisition technology, specifically relating to a time synchronization judgment method for image acquisition of automated operating equipment and a cleaning robot. Background Technology

[0003] Structured light cameras, due to their image acquisition capabilities, are widely integrated into equipment across various industries, such as drones, cleaning robots, autonomous vehicles, and lawnmowers, to enable the corresponding functions of each device.

[0004] Structured light cameras can be triggered to acquire images either externally or internally. External methods involve sending image acquisition signals to the structured light camera via external hardware, while internal methods use a timer mechanism set in internal software to trigger the signal. Automated equipment such as cleaning robots, self-driving cars, and lawnmowers can achieve automatic control based on images acquired by structured light cameras. Therefore, ensuring control is based on images consistent with the real-time position of the automated equipment is crucial.

[0005] Triggering image acquisition signals to the structured light camera through internal or external means may result in missed images or image transmission blockages. This can lead to a situation where the image used by the processing unit to control the automatic operation equipment is out of sync with the current position, resulting in control errors of the automatic operation equipment and affecting its navigation and obstacle avoidance functions.

[0006] To address this technical problem, existing technologies compare the timestamps of the processing unit and the structured light camera to determine whether the image used to control the automated equipment is synchronized with the current position. However, the processing unit and the structured light camera are in different hardware systems, and it is difficult to achieve precise synchronization of their timestamps, resulting in a certain time error. This leads to poor control of the automated equipment by the processing unit. Summary of the Invention

[0007] To address one of the aforementioned technical deficiencies, this application provides a method for determining the time synchronization of image acquisition in automated operation equipment and a cleaning robot. The technical solution includes: On one hand, this application provides a method for determining the time synchronization of image acquisition for an automated operating device, comprising: a first processing unit sending a trigger signal to a structured light camera and generating a first counting identifier for the trigger signal; the structured light camera receiving the trigger signal and acquiring an image of the environment surrounding the automated operating device based on the trigger signal; a second processing unit acquiring the image acquired by the structured light camera and generating a second counting identifier for the image; and the second processing unit determining whether the image acquisition time is synchronized with the trigger signal generation time based on the first counting identifier and the second counting identifier.

[0008] Optionally, the method further includes: if the second processing unit determines that the image acquisition time and the trigger signal generation time are not synchronized, then the first counting identifier and the second counting identifier are reset.

[0009] Optionally, the first counting identifier is determined based on the number of trigger signals sent; the second counting identifier is determined based on the number of images acquired.

[0010] Optionally, the second processing unit determines whether the image acquisition time and the trigger signal generation time are synchronized based on the first counting identifier and the second counting identifier, including: calculating a first difference between the first counting identifier and the second counting identifier; if the first difference is greater than a first threshold, then determining that the image acquisition time and the trigger signal generation time are not synchronized.

[0011] Optionally, the first threshold is determined based on the frame rate of the structured light camera.

[0012] Optionally, resetting the first and second counting identifiers includes: the second processing unit sending a first reset signal to the first processing unit; the first processing unit resetting the first counting identifier based on the first reset signal and sending a second reset signal to the GPIO module of the second processing unit; the GPIO module receiving the second reset signal and resetting the second counting identifier based on the second reset signal.

[0013] Optionally, the method further includes: the first processing unit further generates a first timestamp of the trigger signal; if it is determined that the image acquisition time is synchronized with the trigger signal generation time, the second processing unit uses the first timestamp of the corresponding trigger signal as the timestamp of the image.

[0014] Optionally, the automated operation device further includes a data acquisition unit, and the method further includes: the first processing unit acquiring data acquired by the data acquisition unit and generating a second timestamp of the data, the data including at least the current position of the automated operation device; the second processing unit matching the current position and the image based on the first timestamp and the second timestamp; and the first processing unit controlling the automated operation device to work based on the matched current position and image.

[0015] Optionally, the first processing unit is based on a real-time operating system, and the second processing unit is based on a time-sharing operating system.

[0016] In another aspect, this application provides a cleaning robot, which includes a first processing unit, a structured light camera, and a second processing unit. The structured light camera and the second processing unit are respectively connected to the first processing unit, and the structured light camera is also connected to the second processing unit. The first processing unit is used to send a trigger signal to the structured light camera and generate a first count identifier for the trigger signal. The structured light camera is used to receive the trigger signal and acquire images of the environment surrounding the cleaning robot based on the trigger signal. The second processing unit is used to acquire the images acquired by the structured light camera and generate a second count identifier for the images. The second processing unit is also used to determine whether the image acquisition time is synchronized with the trigger signal generation time based on the first count identifier and the second count identifier.

[0017] Optionally, if the image acquisition time is not synchronized with the trigger signal generation time, the second processing unit and the first processing unit are further configured to: reset the counts of the first counting identifier and the second counting identifier.

[0018] Optionally, the cleaning robot further includes a data acquisition unit connected to the first processing unit; the first processing unit is further configured to: generate a first timestamp of the trigger signal; the second processing unit is further configured to: if it is determined that the image acquisition time is synchronized with the trigger signal generation time, use the first timestamp of the corresponding trigger signal as the timestamp of the image; the first processing unit is further configured to: obtain the current position of the cleaning robot acquired by the data acquisition unit, and generate a second timestamp of the current position; the second processing unit is further configured to: match the current position and the image based on the first timestamp and the second timestamp; the first processing unit is further configured to: control the cleaning robot based on the matched current position and image.

[0019] In another aspect, this application provides a method for determining the time synchronization of image acquisition for an automated operating device. The method includes: acquiring an image of the surrounding environment of the automated operating device acquired by a structured light camera, and generating a second counting identifier for the image; acquiring a first counting identifier; wherein the first counting identifier is generated by a first processing unit based on a trigger signal, the trigger signal being generated by the first processing unit to enable the structured light camera to acquire the image; and determining whether the image acquisition time is synchronized with the trigger signal generation time based on the first counting identifier and the second counting identifier.

[0020] On another front, this application provides a time synchronization determination device for image acquisition of an automated operating device. The device includes: a first acquisition and generation module, configured to acquire an image of the surrounding environment of the automated operating device acquired by a structured light camera, and generate a second counting identifier for the image; a first acquisition module, configured to acquire a first counting identifier; wherein the first counting identifier is generated by a first processing unit based on a trigger signal, the trigger signal being generated by the first processing unit to enable the structured light camera to acquire the image; and a first determination module, configured to determine whether the image acquisition time is synchronized with the trigger signal generation time based on the first counting identifier and the second counting identifier.

[0021] In another aspect, this application provides a method for determining the time synchronization of image acquisition for an automated operating device. The method includes: a structured light camera acquiring an image of the environment surrounding the automated operating device and generating a shooting signal to send to a first processing unit and sending the image to a second processing unit; the first processing unit receiving the shooting signal and generating a third counting identifier for the shooting signal; the second processing unit receiving the image and generating a second counting identifier for the image; and the second processing unit determining whether the image acquisition time and the shooting signal generation time are synchronized based on the third counting identifier and the second counting identifier.

[0022] Optionally, the method further includes: if the second processing unit determines that the image acquisition time and the shooting signal generation time are not synchronized, then the second counting identifier and the third counting identifier are reset.

[0023] Optionally, the third counting identifier is determined based on the number of times the captured signal has been received; the second counting identifier is determined based on the number of images that have been received.

[0024] Optionally, the second processing unit determines whether the image acquisition time and the shooting signal generation time are synchronized based on the second counting identifier and the third counting identifier by: calculating a second difference between the third counting identifier and the second counting identifier; if the second difference is greater than a second threshold, then it is determined that the image acquisition time and the shooting signal generation time are not synchronized.

[0025] Optionally, the second threshold is determined based on the frame rate of the structured light camera.

[0026] Optionally, resetting the count of the second and third count identifiers includes: the second processing unit sending a third reset signal to the first processing unit; the first processing unit resetting the third count identifier based on the third reset signal and sending a second reset signal to the GPIO module of the second processing unit; the GPIO module receiving the second reset signal and resetting the second count identifier based on the second reset signal.

[0027] Optionally, the method further includes: the first processing unit further generates a third timestamp of the shooting signal; if it is determined that the image acquisition time is synchronized with the shooting signal generation time, the second processing unit uses the third timestamp of the corresponding shooting signal as the timestamp of the image.

[0028] Optionally, the automatic operation device further includes a data acquisition unit; the method further includes: the first processing unit acquiring data acquired by the data acquisition unit and generating a second timestamp of the data; the data at least includes the current position of the automatic operation device; the second processing unit matching the current position and the image based on the third timestamp and the second timestamp; the first processing unit controlling the automatic operation device to work based on the matched current position and image.

[0029] Optionally, the first processing unit is based on a real-time operating system, and the second processing unit is based on a time-sharing operating system.

[0030] In another aspect, this application provides a cleaning robot, which includes a first processing unit, a structured light camera, and a second processing unit. The structured light camera and the second processing unit are respectively connected to the first processing unit, and the structured light camera is connected to the second processing unit. The structured light camera is used to: acquire images of the environment surrounding the cleaning robot, generate an image capture signal and send it to the first processing unit, and send the image to the second processing unit. The first processing unit is used to: receive the image capture signal and generate a third counting identifier for the image capture signal. The second processing unit is used to: receive the image and generate a second counting identifier for the image, and determine whether the image acquisition time and the image capture signal generation time are synchronized based on the third counting identifier and the second counting identifier.

[0031] Optionally, if the image acquisition time is not synchronized with the shooting signal generation time, the second processing unit and the first processing unit are further configured to: reset the counts of the third counting identifier and the second counting identifier. Optionally, the cleaning robot further includes a data acquisition unit connected to the first processing unit; the first processing unit is further configured to: generate a third timestamp of the shooting signal; the second processing unit is further configured to: if it is determined that the image acquisition time is synchronized with the shooting signal generation time, use the third timestamp of the corresponding shooting signal as the timestamp of the image; the first processing unit is further configured to: obtain the current position of the cleaning robot acquired by the data acquisition unit, and generate a second timestamp of the current position; the second processing unit is further configured to: match the current position and the image based on the third timestamp and the second timestamp; the first processing unit is further configured to: control the cleaning robot based on the matched current position and image.

[0032] In another aspect, this application provides a method for determining the time synchronization of image acquisition for an automated operating device. The method includes: acquiring an image of the surrounding environment of the automated operating device captured by a structured light camera, and generating a second counting identifier for the image; acquiring a third counting identifier; wherein the third counting identifier is generated by a first processing unit based on a shooting signal, and the shooting signal is generated by the structured light camera based on the image acquisition action; and determining whether the image acquisition time and the shooting signal generation time are synchronized based on the second counting identifier and the third counting identifier.

[0033] On another front, this application provides a time synchronization determination device for image acquisition of an automated operating device. The device includes: a second acquisition and generation module, used to acquire an image of the surrounding environment of the automated operating device captured by a structured light camera, and generate a second counting identifier for the image; a second acquisition module, used to acquire a third counting identifier; wherein the third counting identifier is generated by a first processing unit based on a shooting signal, and the shooting signal is generated by the structured light camera based on the image acquisition action; and a second determination module, used to determine whether the image acquisition time and the shooting signal generation time are synchronized based on the second counting identifier and the third counting identifier.

[0034] The present application provides a time synchronization determination method for image acquisition in an automated operating device. This method generates a first count identifier for a trigger signal and a second count identifier for the image. Based on the first and second count identifiers, it determines whether the image acquisition time is synchronized with the trigger signal generation time. Compared to using the timestamps of the structured light camera and the first processing unit for determination, this method avoids the drawback of difficulty in accurately synchronizing the time of the structured light camera and the first processing unit, thereby improving the matching accuracy between the image acquisition time and the trigger signal generation time.

[0035] This application provides another method for determining the time synchronization of image acquisition in an automated operating device. It generates a third counting identifier for the capturing signal and a second counting identifier for the image, and determines whether the image acquisition time is synchronized with the capturing signal generation time based on the third and second counting identifiers. Compared to using the timestamps of the structured light camera and the first processing unit for determination, this method avoids the drawback of difficulty in accurately synchronizing the time of the structured light camera and the first processing unit, thereby improving the matching accuracy between the image acquisition time and the trigger signal generation time. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0037] Figure 1 This is a schematic diagram of the structure of a cleaning robot provided in one embodiment of this application.

[0038] Figure 2 This is one of the interactive flowcharts of the image acquisition time synchronization judgment method provided in this embodiment.

[0039] Figure 3 A flowchart of a time synchronization determination method for image acquisition of an automated operating device provided in one embodiment of this application; Figure 4 This is another interactive flowchart of the image acquisition time synchronization judgment method provided in this embodiment; Figure 5 Another flowchart of a time synchronization determination method for image acquisition of an automated operating device provided in another embodiment of this application; Figure 6 A schematic diagram of a time synchronization determination device for image acquisition of an automatic operating device provided in one embodiment of this application; Figure 7 This is a schematic diagram of the structure of a time synchronization judgment device for automatic operation equipment image acquisition provided in another embodiment of this application.

[0040] In the figure, 110 is the first processing unit; 120 is the structured light camera; 130 is the second processing unit; 140 is the data acquisition unit; 150 is the walking component; 610 is the first acquisition and generation module; 620 is the first acquisition module; 630 is the first judgment module; 710 is the second acquisition and generation module; 720 is the second acquisition module; and 730 is the second judgment module. Detailed Implementation

[0041] The technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. This application will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other. It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0042] The image acquisition time synchronization judgment method provided in this application can be applied to automated operating equipment. Automated operating equipment can be mobile automated operating equipment based on image acquisition, such as cleaning robots (e.g., sweeping robots, window cleaning robots, etc.), autonomous vehicles, food delivery robots, and automatic lawnmowers.

[0043] like Figure 1 As shown, the automated operating device (hereinafter, taking a cleaning robot as an example) may include a first processing unit 110, a structured light camera 120, a second processing unit 130, and a data acquisition unit 140. The structured light camera 120, the second processing unit 130, and the data acquisition unit are respectively connected to the first processing unit 110 to realize data interaction; the structured light camera 120 and the data acquisition unit 140 are respectively connected to the second processing unit 130 to realize data interaction.

[0044] To ensure the timeliness of data and / or signal transmission, the structured light camera 120, the second processing unit 130, and the data acquisition unit 140 can be wired to the first processing unit 110, and the structured light camera 120 and the data acquisition unit 140 can also be wired to the second processing unit 130. In some embodiments, to simplify the structure of the automated operating device, the structured light camera 120, the second processing unit 130, and the data acquisition unit 140 can be wirelessly connected to the first processing unit 110, and the structured light camera 120 and the data acquisition unit 140 can be wirelessly connected to the second processing unit 130.

[0045] For example, such as Figure 1 As shown, the structured light camera 120, the second processing unit 130, and the data acquisition unit 140 can be connected to the first processing unit 110 via wired connections such as a serial interface, USB interface, Ethernet interface, or Serial Peripheral Interface (SPI). The structured light camera 120 and the data acquisition unit 140 can be connected to the second processing unit 130 via wired connections such as a serial interface, USB interface, Ethernet interface, or Serial Peripheral Interface.

[0046] The data acquisition unit 140 is a device for collecting data on the automated operating equipment and its surrounding environment. The data acquisition unit 140 may include devices for collecting environmental data, such as temperature sensors, humidity sensors, and radar. The temperature sensor can be used to obtain the temperature of the environment in which the cleaning robot is located, the humidity sensor can be used to obtain the humidity of the environment in which the cleaning robot is located, and the radar can be used to obtain the distance between obstacles in the environment and the cleaning robot. The data acquisition unit 140 may also include devices for collecting the attitude of the cleaning robot, such as a three-axis gyroscope, a three-axis accelerometer, and a three-axis electronic compass.

[0047] The data acquisition unit may also include a positioning component, which can acquire the location of the cleaning robot. The positioning component may include an outdoor locator (e.g., an LBS (Location Based Services) locator, a GPS locator, etc.) and / or an indoor locator (e.g., a Bluetooth locator, a Wi-Fi locator, an RFID locator, etc.).

[0048] The first processing unit 110 can be used to generate timestamps for the data acquired by the data acquisition unit 140. To ensure the real-time nature of the generated timestamps and the triggering of image acquisition by the structured light camera 120, the first processing unit 110 can be embedded with a real-time operating system (e.g., FreeRTOS, RT-Thread, ThreadX, embOS, etc.). A real-time operating system (RTOS) is an operating system that can accept and process external events or data at a sufficiently fast speed, and whose processing results can control the production process or respond quickly to the processing system within a specified time, scheduling all available resources to complete real-time tasks and controlling all real-time tasks to run in a coordinated manner. Its main characteristics are timely response and high reliability. For example, the first processing unit 110 can be a microcontroller unit (MCU) embedded with a FreeRTOS system.

[0049] The structured light camera 120 is a device for acquiring images of the environment in which a cleaning robot is located. The structured light camera 120 may include a projector and a camera. The projector can actively emit infrared light, invisible to the naked eye, into the environment in which the cleaning robot is located. The camera is used to capture structured light images of the environment in which the projected infrared light is located. The pattern of the structured light emitted by the projector can be in various forms, such as crosshairs, grids, etc.

[0050] To avoid interference from highly reflective obstacles, the projector can include multiple projection devices. The projection devices are used to emit a single strip of structured light into the environment where the cleaning robot is located. The multiple projection devices are set at different positions of the cleaning robot. For example, a projection device is set on the left side of the cleaning robot (also called the left projection device), and a projection device is set on the right side of the cleaning robot (also called the right projection device).

[0051] During image acquisition, one of the multiple projection devices can be controlled to emit a single strip of structured light into the environment where the cleaning robot is located, according to a preset emission sequence, while the other projection devices are controlled not to emit infrared light.

[0052] As an example only, the preset transmission sequence can be left projection device - no projection device - right projection device - no projection device. The left projection device is turned on to capture the first frame of structured light image, while the left and right projection devices are turned off to capture the second frame of structured light image. Then the right projection device is turned on to capture the third frame of structured light image, and finally the left and right projection devices are turned off to capture the fourth frame of structured light image. The shooting is repeated according to the preset transmission sequence.

[0053] The structured light camera 120 can actively capture structured light images of the environment in which the cleaning robot is located. During the operation of the cleaning robot, the structured light camera 120 can acquire images at a preset frame rate (e.g., 75 Hz), where the frame rate represents the number of times the structured light camera 120 acquires images per unit time.

[0054] For each frame of image acquired, the structured light camera 120 can send a capture signal to the first processing unit 110. The first processing unit 110 can generate a count identifier and a timestamp for the capture signal. The count identifier represents the number of capture signals received by the first processing unit 110 from the structured light camera 120. After each image acquisition is completed, the structured light camera 120 can send a capture signal to the first processing unit 110, which can receive the capture signal and generate a count identifier for it.

[0055] The structured light camera 120 can also passively trigger the acquisition of structured light images of the cleaning robot's environment. The first processing unit 110 generates a trigger signal and a count of the trigger signal. The first processing unit 110 sends the trigger signal to the structured light camera 120. Upon receiving the trigger signal, the structured light camera 120 can acquire one frame of image based on the trigger signal. The trigger signal can be a high-level signal. Each time the first processing unit 110 generates and sends a trigger signal to the structured light camera 120, it generates a count of the trigger signal. Each time the structured light camera 120 receives a trigger signal, it acquires one frame of image of the cleaning robot's surroundings.

[0056] The second processing unit 130 can be a data processing device. The second processing unit 130 can acquire images captured by the structured light camera 120 and generate image count markers. The second processing unit 130 can determine whether the image acquisition time is synchronized with the trigger signal generation time based on the trigger signal count markers and the image count markers. The second processing unit 130 can also determine whether the image acquisition time is synchronized with the image generation time based on the shooting signal count markers and the image count markers.

[0057] Once it is determined that the image acquisition time is synchronized with the trigger signal generation time, or that the image acquisition time is synchronized with the shooting signal generation time, the first processing unit 110 can use the image to control the automatic operation equipment, such as controlling the automatic operation equipment to avoid obstacles, switch working modes, etc.

[0058] When it is determined that the image acquisition time and the trigger signal generation time are out of sync, the count flags for both the trigger signal and the image are reset; similarly, when the image acquisition time and the shooting signal generation time are out of sync, the count flags for both the image and the shooting signal are reset. In other words, a missynchronization between the image acquisition time and the trigger signal generation time, or between the image acquisition time and the shooting signal generation time, indicates a fault such as missed images or image transmission blockage during data transmission. Using images with data transmission faults to control automated equipment may lead to control errors, affecting the navigation and obstacle avoidance capabilities of the automated equipment.

[0059] To improve the accuracy of data processing, the second processing unit 130 may include an ARM chip based on a time-sharing operating system.

[0060] like Figure 1 As shown, the cleaning robot may also include a walking assembly 150, which can drive the cleaning robot to move. The walking assembly 150 may include a drive mechanism and a walking mechanism. The drive mechanism provides a power source to drive the walking mechanism, which moves under the drive of the drive mechanism to drive the cleaning robot to move.

[0061] When the second processing unit 130 determines that the image acquisition time is synchronized with the trigger signal generation time or the image acquisition time is synchronized with the shooting signal generation time, the second processing unit 130 can match the data of the automatic operation device collected by the data acquisition unit according to the timestamp of the image, and control the automatic operation device to work using the matched image and the data of the automatic operation device.

[0062] It is understandable that the second processing unit 130 based on the time-sharing operating system has stronger computing power than the first processing unit 110 based on the real-time operating system. It is more timely than controlling the walking component 150 by the first processing unit 110 alone, and can achieve more precise control of the walking component 150.

[0063] In some embodiments, the cleaning robot may further include one or more peripheral device interfaces and at least one peripheral device. The processor, memory, and peripheral device interfaces can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface via a bus, signal line, or circuit board. Indicatively, peripheral devices include, but are not limited to, radio frequency circuits, touch displays, audio circuits, and power supplies. Of course, the cleaning robot may also include fewer or more components, and this embodiment does not limit this.

[0064] Understandably, cleaning robots are devices used for cleaning and sanitation work. Depending on their application and purpose, cleaning robots can be divided into various types, such as sweeping robots, window cleaning robots, and disinfection robots.

[0065] This embodiment provides a method for determining the time synchronization of image acquisition in an automated operating device. Figure 2 This is an interactive flowchart of one of the image acquisition time synchronization determination methods provided in this embodiment. This method can be applied to the aforementioned automated motion device, which includes a first processing unit, a structured light camera, and a second processing unit. Although the process described below includes multiple operations appearing in a specific order, it should be clearly understood that these processes may also include more or fewer operations, which can be executed sequentially or in parallel (e.g., using a parallel processor or a multi-threaded environment). Figure 2 As shown, the method includes the following steps: (S210~S240): S210, the first processing unit sends a trigger signal to the structured light camera and generates a first count identifier for the trigger signal.

[0066] The trigger signal can be generated by the first processing unit. This trigger signal is used to cause the structured light camera to acquire images; that is, when the structured light camera receives the trigger signal, it acquires images of the surrounding area of ​​the automatically operating device. Figure 1As shown, the first processing unit 110 can send a trigger signal to the structured light camera 120 via a bus and / or signal lines. In an automatically operating device (e.g., Figure 1 During the operation of the cleaning robot shown, the first processing unit 110 can send trigger signals to the structured light camera 120 at a preset frequency (e.g., 13.3ms / time).

[0067] The first processing unit 110 can determine a first count identifier for the trigger signal based on the number of times the trigger signal has been sent. The first count identifier can be in various forms, such as English letters, numbers, or Chinese characters. For example, if the first processing unit 110 has already sent 12 trigger signals when sending the current trigger signal, then the first count identifier for the current trigger signal can be "13". It is understandable that the first processing unit 110 can determine the first count identifier for the trigger signal relatively accurately based on the number of times the trigger signal has been sent.

[0068] When the first processing unit 110 generates the first count identifier of the trigger signal, it may also generate the first timestamp of the trigger signal. For example, the first processing unit 110 may use the time when the trigger signal is generated as the first timestamp of the trigger signal.

[0069] When the first processing unit 110 sends a trigger signal to the structured light camera 120, it can also send the first count identifier and / or the first timestamp of the trigger signal to the second processing unit 130 via a bus and / or signal line.

[0070] S220: The structured light camera receives a trigger signal and acquires images of the environment surrounding the automatically operating equipment based on the trigger signal.

[0071] Each time a structured light camera receives a trigger signal, it can capture one frame of the automatically running device (e.g., Figure 1 Images of the environment surrounding the cleaning robot shown.

[0072] S230, the second processing unit acquires the image captured by the structured light camera and generates a second counting identifier for the image.

[0073] like Figure 1 As shown, after the structured light camera 120 acquires a frame image based on the trigger signal, it can send the frame image to the second processing unit 130 via a bus and / or signal line. After acquiring the frame image, the second processing unit 130 can generate a second count identifier for that frame image. That is, the second count identifier can be determined based on the number of images acquired. The second processing unit generates a second count identifier corresponding to each frame image it receives.

[0074] For example, if the second processing unit 130 has acquired 25 images when acquiring images, then the second count identifier for that frame image is "26". It is understandable that the second processing unit 130 can determine the second count identifier relatively accurately based on the number of images received.

[0075] S240, the second processing unit determines whether the image acquisition time and the trigger signal generation time are synchronized based on the first counting identifier and the second counting identifier.

[0076] The error between the first and second counting identifiers determines the accuracy of subsequent control of the automatic operating equipment. To ensure the accuracy of the automatic operating equipment, if the first and second counting identifiers are inconsistent, it is determined that the image acquisition time and the trigger signal generation time are out of sync.

[0077] In practice, to ensure the normal operation of the automated equipment, errors between the first and second counting identifiers are permissible. The second processing unit can calculate a first difference between the first and second counting identifiers and determine whether the absolute value of the first difference is greater than a first threshold, thereby determining whether the image acquisition time and the trigger signal generation time are synchronized. The first threshold represents the range within which errors between the first and second counting identifiers are permissible.

[0078] For example, the first threshold can be 10, 15, or 20, etc. Those skilled in the art can make a reasonable choice based on the actual situation, and no limitation is made here.

[0079] If the absolute value of the first difference is greater than the first threshold, the second processing unit can determine that the image acquisition time and the trigger signal generation time are out of sync; if the absolute value of the first difference is less than or equal to the first threshold, the second processing unit determines that the image acquisition time and the trigger signal generation time are synchronized.

[0080] By setting a first threshold, when the first and second count identifiers are inconsistent, it is determined that the image acquisition time and the trigger signal generation time are out of sync. This provides an appropriate fault tolerance rate for the synchronization of the image acquisition time and the trigger signal generation time, thus providing a certain fault tolerance rate for hardware failures in the automatic operation equipment during the image acquisition process. This avoids the automatic operation equipment from frequently performing synchronization operations between the image acquisition time and the trigger signal generation time, thereby avoiding over-synchronization that could cause the automatic operation equipment to malfunction.

[0081] For example, if the first threshold is 20, the first count flag of the trigger signal is 45, and the second count flag determined by the received image is 10, then the first difference between the first count flag and the second count flag is greater than the first threshold, and it is determined that the image acquisition time and the trigger signal generation time are out of sync.

[0082] In the above embodiments, a first counting identifier for the trigger signal and a second counting identifier for the image are generated. Based on the first and second counting identifiers, it is determined whether the image acquisition time and the trigger signal generation time are synchronized. Compared to using the timestamps of the structured light camera and the first processing unit for judgment, this method avoids the drawback of the difficulty in accurately synchronizing the time of the structured light camera and the first processing unit, thereby improving the matching accuracy between the image acquisition time and the trigger signal generation time.

[0083] The first threshold can be set based on the frame rate of the structured light camera, where the frame rate represents the number of times the structured light camera acquires images per unit time. When the structured light camera acquires images under the trigger of the first processing unit, the frame rate of the structured light camera is related to the frequency at which the first processing unit sends the trigger signal; that is, the first threshold can be related to the frequency at which the first processing unit sends the trigger signal. Determining the first threshold based on the frame rate of the structured light camera can increase the rationality of the first threshold setting.

[0084] The higher the frame rate of the structured light camera or the frequency at which the first processing unit sends trigger signals, the larger the first threshold. For example, when the frame rate of the structured light camera is 60Hz, the first threshold can be 25; when the frame rate of the structured light camera is 75Hz, the first threshold can be 30.

[0085] If the second processing unit determines that the image acquisition time and the trigger signal generation time are not synchronized, it resets the first and second counters. That is, after the first and second counters are reset, the first processing unit regenerates and sends the trigger signal, and restarts counting on the first counter. The second processing unit then receives the image again and restarts counting on the second counter. Based on the new first and second counters, the second processing unit re-determines whether the image acquisition time and the trigger signal generation time are synchronized.

[0086] Resetting the first counter flag involves clearing the count of trigger signals already sent by the first processing unit to zero. For example, after the counter reset is completed, when the first processing unit sends the next trigger signal, the first counter flag for that next trigger signal can be "1".

[0087] Resetting the second counter identifier involves clearing the number of images received by the second processing unit to zero. For example, after resetting the counter, when the second processing unit receives the next image from the structured light camera, the second counter identifier for that image can be "1".

[0088] In one or more embodiments, in order to improve the timeliness of the count reset, the second processing unit may send a first reset signal to the first processing unit, the first processing unit resets the first count flag based on the reset signal, and sends a second reset signal to the GPIO module of the second processing unit, the GPIO module receives the second reset signal and resets the second count flag.

[0089] The second processing unit can send a first reset signal to the first processing unit after receiving an interrupt request from the first processing unit. Sending the first reset signal passively ensures the efficiency of the main task while also promptly handling important device interrupt requests. Alternatively, the second processing unit can also actively send the first reset signal if it determines that the image acquisition time and the trigger signal generation time are out of sync.

[0090] The GPIO (General-purpose input / output) module can generate GPIO interrupts, including edge-triggered interrupts and level-to-toggle interrupts. When the first processing unit sends a second reset signal to the GPIO module, the GPIO module detects a pin level change and, if the interrupt trigger condition is met, triggers a GPIO interrupt. The second processing unit jumps to the interrupt handling address to handle the interrupt. To avoid corrupting the main task data, the second processing unit saves the relevant registers and enters the interrupt service function. After executing the interrupt service function, the second processing unit restores the relevant registers and returns to the main task to continue program execution. Compared to resetting the second counter flag through software, resetting the second counter flag through the GPIO module offers higher real-time performance.

[0091] In one or more embodiments, the first processing unit may further generate a first timestamp of the trigger signal, and if it is determined that the image acquisition time is synchronized with the trigger signal generation time, the second processing unit shall use the first timestamp of the corresponding trigger signal as the timestamp of the image.

[0092] The second processing unit can cache the first count identifiers of multiple historical trigger signals and their corresponding first timestamps. For example, if the first count identifier of the current trigger signal is "30", then the second processing unit can cache the count identifiers of historical trigger signals with first count identifiers from "10" to "29" and their corresponding first timestamps.

[0093] If the image acquisition time is determined to be synchronized with the trigger signal generation time, the second processing unit can determine whether the second counting identifier of the current image and the first counting identifier of the current trigger signal are consistent. If the second counting identifier of the current image and the first counting identifier of the current trigger signal are consistent, the second processing unit can use the first timestamp of the current trigger signal as the timestamp of the current image. If the second counting identifier of the current image and the first counting identifier of the current trigger signal are inconsistent, the second processing unit can determine a historical trigger signal that is consistent with the second counting identifier of the current image based on the second counting identifier of the current image, and use the first timestamp of the historical trigger signal as the timestamp of the current image.

[0094] As an example only, if the second count identifier of the current image is "29" and the first count identifier of the current trigger signal is "30", based on the first count identifier of the trigger signal and the second count identifier of the image, it can be determined that the image acquisition time is synchronized with the trigger signal generation time. However, the second count identifier of the current image and the first count identifier of the current trigger signal are not consistent. Therefore, the second processing unit cannot directly use the first timestamp of the current trigger signal as the timestamp of the current image. The second processing unit can use the second count identifier of the current image (i.e., "29") to find the historical trigger signal with the first count identifier "29" from the first count identifiers of multiple cached historical trigger signals, and use the first timestamp of the historical trigger signal with the first count identifier "29" as the timestamp of the current image.

[0095] The first processing unit can acquire data from the automatically operating device and generate a second timestamp for the data. The data from the automatically operating device can be collected by a data acquisition unit, which can be integrated into the automatically operating device. The data from the automatically operating device includes at least its current location. Figure 1 As shown, the first processing unit 110 can acquire data from the data acquisition unit of the automatically running device through the bus and / or signal line, and the first processing unit 110 can generate a second timestamp of the data based on the time of acquiring the data of the automatically running device.

[0096] The second processing unit can match the current location with the image based on the first and second timestamps. The first processing unit then controls the automatic operation device to work based on the matched current location and image.

[0097] For example, the second processing unit can calculate the time difference between the second timestamp of the current location and the first timestamp of the image. If the time difference between the second timestamp of the current location and the first timestamp of the image is less than a preset time threshold, the second processing unit can determine that the current location matches the image.

[0098] The second processing unit can process the matched current location and image to obtain information related to the operation of the automated equipment. For example, based on the matched current location and image, the second processing unit can determine the positions of obstacles near the current location of the automated equipment and the distance between each obstacle and the cleaning robot, and generate a walking path to avoid the obstacles. As another example, based on the matched current location and image, the second processing unit can determine whether there is any trash that needs to be cleaned near the current location of the cleaning robot, and generate a walking path to clean up the trash starting from the current location.

[0099] The second processing unit can control the operation of the automated equipment through the first processing unit based on information related to the operation of the automated equipment obtained after processing the matched current location and image. For example, the second processing unit can generate a walking path for avoiding obstacles near the current location of the automated equipment or for cleaning up garbage near the current location of the automated equipment based on the matched current location and image, and send it to the first processing unit. The first processing unit can generate a sequence of control commands based on the walking path and send the sequence of control commands to the walking component of the cleaning robot to control the walking component of the cleaning robot to walk according to the walking path.

[0100] In summary, the time synchronization determination method for image acquisition of automated operating equipment provided in this application involves a first processing unit sending a trigger signal to a structured light camera and generating a first count identifier for the trigger signal; the structured light camera receiving the trigger signal and acquiring images of the surrounding environment of the automated operating equipment based on the trigger signal; a second processing unit acquiring the images acquired by the structured light camera and generating a second count identifier for the images; and the second processing unit determining whether the image acquisition time is synchronized with the trigger signal generation time based on the first and second count identifiers. Compared to using the time of the structured light camera and the first processing unit itself, determining whether the image acquisition time is synchronized with the trigger signal generation time based on the first and second count identifiers can be done quickly and accurately.

[0101] For example, if the first threshold is set to 7, and the first processing unit sends the 67th trigger signal, under normal circumstances, the structured light camera should perform the 67th image acquisition. However, if the structured light camera has a missed image malfunction and only performs the 55th image acquisition, only 55 images are transmitted to the second processing unit, then the second count identifier of the current image generated by the second processing unit is "55", which is inconsistent with the actual second count identifier of the current image, "67". The second processing unit compares the second count identifier of the current image (i.e., "55") with the first count identifier of the current trigger signal (i.e., "67"), which can quickly and accurately determine that the image acquisition time and the trigger signal generation time are out of sync.

[0102] For example, if the first threshold is set to 10, the first processing unit sends the 67th trigger signal, and the structured light camera performs the 67th image acquisition. Due to a data transmission failure, the second processing unit only receives the first 55 images. The second count identifier of the current image generated by the second processing unit is "55", which is inconsistent with the actual second count identifier of the current image, "67". The second processing unit compares the second count identifier of the current image (i.e., "55") with the first count identifier of the current trigger signal (i.e., "67") to quickly and accurately determine that the image acquisition time and the trigger signal generation time are out of sync.

[0103] When the second processing unit determines that the image acquisition time and the trigger signal generation time are out of sync, it resets the first and second counting flags to restore synchronization between the image acquisition time and the trigger signal generation time, which can effectively avoid the accumulation of errors.

[0104] The first processing unit can acquire data from the automatically running device and generate a second timestamp for the data. When it is determined that the image acquisition time is synchronized with the trigger signal generation time, the second processing unit uses the first timestamp of the corresponding trigger signal as the timestamp of the image. This ensures that the timestamps of the data and images from the automatically running device used for matching are both generated by the first processing unit based on the real-time operating system. This guarantees the timeliness of the timestamps while avoiding the generation of timestamps for the data and images from the automatically running device through different hardware systems. This would lead to a certain time error, making it impossible to match the data and images from the automatically running device, or reducing the accuracy of the matching. Consequently, the control capabilities of the second and first processing units over the automatically running device would be poor.

[0105] Figure 3 This is a flowchart of a time synchronization determination method for image acquisition in an automated operating device according to an embodiment of this application. This method can be applied to a second processing unit in the aforementioned automated operating device. Although the process described below includes multiple operations appearing in a specific order, it should be clearly understood that these processes may also include more or fewer operations, which can be executed sequentially or in parallel (e.g., using a parallel processor or a multi-threaded environment). Figure 3 As shown, the method includes the following steps: (S310~S330): S310, the second processing unit acquires an image of the environment surrounding the automated operating device captured by the structured light camera, and generates a second counting identifier for the image.

[0106] like Figure 1As shown, the second processing unit 130 can acquire images of the surrounding environment of the automated operating device from the structured light camera 120 via a bus and / or signal lines. After acquiring the images of the surrounding environment of the automated operating device, the second processing unit 130 can generate a second counting identifier for that frame image. For details, please refer to the foregoing description, which will not be repeated here.

[0107] S320, the second processing unit acquires the first counting identifier.

[0108] like Figure 1 As shown, the first counting identifier can be generated by the first processing unit 110 based on a trigger signal. The trigger signal can be generated by the first processing unit 110 and can be used to cause the structured light camera 120 to acquire images. For details, please refer to the foregoing description, which will not be repeated here.

[0109] S330, the second processing unit determines whether the image acquisition time and the trigger signal generation time are synchronized based on the first and second counting identifiers.

[0110] The second processing unit can calculate the first difference between the first counting identifier and the second counting identifier, and determine whether the absolute value of the first difference is greater than a first threshold, thereby determining whether the image acquisition time and the trigger signal generation time are synchronized. For details, please refer to the aforementioned description, which will not be repeated here.

[0111] Figure 6 This is a schematic diagram of a time synchronization determination device for image acquisition in an automatic operating device according to an embodiment of this application. This time synchronization determination device for image acquisition in an automatic operating device can be applied to the second processing unit in the aforementioned automatic operating device, such as... Figure 6 As shown, the time synchronization determination device for image acquisition in this automatic operation equipment may include: The first acquisition and generation module 610 is used to acquire images of the surrounding environment of the automatically operating equipment collected by the structured light camera and generate a second counting identifier for the image; for details, please refer to the relevant description above, which will not be repeated here.

[0112] The first acquisition module 620 is used to acquire a first count identifier; wherein the first count identifier is generated by the first processing unit based on a trigger signal, and the trigger signal is generated by the first processing unit to enable the structured light camera to acquire an image; for details, please refer to the relevant description above, which will not be repeated here.

[0113] The first judgment module 630 is used to determine whether the image acquisition time and the trigger signal generation time are synchronized based on the first and second counting identifiers. For details, please refer to the aforementioned description, which will not be repeated here.

[0114] This embodiment provides a method for determining the time synchronization of image acquisition in an automated operating device. Figure 4 This is another interactive flowchart of the image acquisition time synchronization determination method provided in this embodiment. This method can be applied to the aforementioned automated motion device, which includes a first processing unit, a structured light camera, and a second processing unit. Although the process described below includes multiple operations appearing in a specific order, it should be clearly understood that these processes may also include more or fewer operations, which can be executed sequentially or in parallel (e.g., using a parallel processor or a multi-threaded environment). Figure 4 As shown, the method includes the following steps: (S410~S460): The S410 is a structured light camera that captures images of the environment surrounding automated equipment.

[0115] In automated operating equipment (such as Figure 1 During the operation of the cleaning robot shown, the structured light camera can automatically acquire images of the environment around the automatically operating equipment at a preset frame rate (e.g., 75 Hz).

[0116] S420, the structured light camera generates an image signal and sends it to the first processing unit.

[0117] like Figure 1 As shown, for each frame of image acquired, the structured light camera 120 can generate a capture signal and send the capture signal to the first processing unit 110 via a bus and / or signal line.

[0118] S430, the structured light camera sends the image to the second processing unit.

[0119] like Figure 1 As shown, the structured light camera 120 can send the acquired images to the second processing unit 130 via a bus and / or signal line.

[0120] S440, the first processing unit receives the shooting signal and generates a third counting identifier for the shooting signal.

[0121] like Figure 1 As shown, after receiving the shooting signal, the first processing unit 110 can generate a third count identifier for the shooting signal corresponding to the image. The third count identifier for the shooting signal can be determined based on the number of shooting signals already received by the first processing unit 110. As an example only, if the first processing unit 110 has completed the reception of 34 shooting signals when receiving the current shooting signal, then the third count identifier for the current shooting signal can be "35".

[0122] The first processing unit 110 can also generate a third timestamp of the captured signal. By way of example only, the first processing unit 110 can generate a third timestamp based on the time the captured signal is received.

[0123] S450, the second processing unit receives the image and generates a second counting identifier for the image.

[0124] The second processing unit can determine a second counting identifier for the image based on the number of received images. For details, please refer to the foregoing description; further elaboration will not be repeated here.

[0125] S460, the second processing unit determines whether the image acquisition time and the shooting signal generation time are synchronized based on the third counting identifier and the second counting identifier.

[0126] The error between the second and third counting markers determines the accuracy of subsequent control of the automatic operating equipment. To ensure the accuracy of the automatic operating equipment, if the third and second counting markers are inconsistent, it can be determined that the image acquisition time and the shooting signal generation time are out of sync.

[0127] In practical operation, to ensure the normal operation of the automated equipment, errors between the second and third counting markers are permissible. The second processing unit can calculate the second difference between the third and second counting markers and determine whether the absolute value of the second difference is greater than a second threshold, thereby determining whether the image acquisition time and the shooting signal generation time are synchronized. The second threshold represents the permissible range of error between the second and third counting markers. For example, the second threshold can be 10, 20, or 30, etc.

[0128] When the absolute value of the second difference is greater than the second threshold, the second processing unit can determine that the image acquisition time and the shooting signal generation time are out of sync; when the absolute value of the second difference is less than or equal to the second threshold, the second processing unit determines that the image acquisition time and the shooting signal generation time are synchronized.

[0129] By setting a second threshold, when the second and third counting identifiers are inconsistent, it is determined that the image acquisition time and the shooting signal generation time are out of sync. This provides an appropriate fault tolerance rate for the synchronization of the image acquisition time and the shooting signal generation time, thus providing a certain fault tolerance rate for hardware failures in the automatic operation equipment during the image acquisition process. This avoids the automatic operation equipment from frequently performing the operation of synchronizing the image acquisition time and the shooting signal generation time, thereby avoiding over-synchronization.

[0130] For example, if the second threshold is 20, the third count identifier of the captured signal is 15, and the second count identifier of the image acquired based on the captured signal is 45, then the absolute value of the second difference between the third count identifier of the captured signal and the second count identifier of the image is greater than the second threshold. Therefore, the second processing unit can determine that the image acquisition time and the captured signal generation time are not synchronized.

[0131] In the above embodiments, a third counting identifier for the capturing signal and a second counting identifier for the image are generated. Based on the third and second counting identifiers, it is determined whether the image acquisition time and the capturing signal generation time are synchronized. Compared to using the timestamps of the structured light camera and the first processing unit for judgment, this method avoids the drawback of the structured light camera and the first processing unit's time being difficult to synchronize precisely, thereby improving the matching accuracy between the image acquisition time and the trigger signal generation time.

[0132] The second threshold can be set based on the frame rate of the structured light camera, where the frame rate represents the number of times the structured light camera acquires images per unit time. The higher the frame rate of the structured light camera, the larger the second threshold. Determining the second threshold based on the frame rate of the structured light camera increases the rationality of the second threshold setting.

[0133] For example, when the frame rate of the structured light camera is 60Hz, the second threshold can be 25; when the frame rate of the structured light camera is 75Hz, the second threshold can be 30.

[0134] If the second processing unit determines that the image acquisition time and the shooting signal generation time are not synchronized, it resets the second and third counters. That is, after the third and second counters are reset, the structured light camera re-captures images of the area around the automated equipment and sends the shooting signal to the first processing unit. The first processing unit restarts counting on the third counter, and the second processing unit receives the images again and restarts counting on the second counter. Based on the new third and second counters, the second processing unit re-determines whether the image acquisition time and the shooting signal generation time are synchronized.

[0135] Resetting the third counter flag involves clearing the count of the shooting signals already received by the first processing unit to zero. For example, after the counter reset is completed, when the first processing unit receives the next shooting signal from the structured light camera, the third counter flag for that next shooting signal can be "1".

[0136] Resetting the second counter identifier involves clearing the number of images received by the second processing unit to zero. For example, after resetting the counter, when the second processing unit receives the next image from the structured light camera, the second counter identifier for that image can be "1".

[0137] In one or more embodiments, in order to improve the timeliness of the count reset, the second processing unit may send a third reset signal to the first processing unit; the first processing unit resets the third count identifier based on the third reset signal and sends a second reset signal to the GPIO module of the second processing unit; the GPIO module receives the second reset signal and resets the second count identifier based on the second reset signal.

[0138] The second processing unit can send a third reset signal to the first processing unit after receiving an interrupt request from the first processing unit. Sending the third reset signal passively ensures the efficiency of the main task while also promptly handling important device interrupt requests. Alternatively, the second processing unit can also actively send a third reset signal if it determines that the image acquisition time and the shooting signal generation time are out of sync.

[0139] Compared to resetting the second counter flag via software, resetting the second counter flag via the GPIO module offers higher real-time performance. For details, please refer to the aforementioned descriptions, which will not be repeated here.

[0140] In one or more embodiments, the first processing unit may further generate a third timestamp of the shooting signal, and if it is determined that the image acquisition time is synchronized with the shooting signal generation time, the second processing unit 130 uses the third timestamp of the corresponding shooting signal as the timestamp of the image.

[0141] The second processing unit can cache the third counter identifiers and their corresponding third timestamps of multiple historical shooting signals. For example, if the third counter identifier of the current shooting signal is "30", then the second processing unit can cache the third counter identifiers and their corresponding third timestamps of historical shooting signals with third counter identifiers from "10" to "29".

[0142] If it is determined that the image acquisition time is synchronized with the shooting signal generation time, the second processing unit 130 can determine whether the second counting identifier of the current image and the third counting identifier of the current shooting signal are consistent. If the second counting identifier of the current image and the third counting identifier of the current shooting signal are consistent, the second processing unit can use the third timestamp of the current shooting signal as the timestamp of the current image.

[0143] If the second counting identifier of the current image and the third counting identifier of the current shooting signal are inconsistent, the second processing unit can determine the historical shooting signal that is consistent with the second counting identifier of the current image based on the second counting identifier of the current image, and use the third timestamp of the historical shooting signal as the timestamp of the current image.

[0144] The first processing unit can receive the collected data from the automated operating equipment and generate a second timestamp for the data. For details, please refer to the aforementioned descriptions, which will not be repeated here.

[0145] The second processing unit can match the current location with the image based on the second and third timestamps of the data.

[0146] For example, the second processing unit can calculate the time difference between the second timestamp of the current location and the third timestamp of the image. If the time difference between the second timestamp of the current location and the third timestamp of the image is less than a preset time threshold, the second processing unit can determine that the current location matches the image.

[0147] The first processing unit controls the automatic operation of the device based on the matched current location and image. For details, please refer to the aforementioned descriptions, which will not be repeated here.

[0148] Figure 5 This is another flowchart illustrating a time synchronization determination method for image acquisition in an automated operating device, provided in another embodiment of this application. This method can be applied to the second processing unit in the aforementioned automated operating device. Although the process described below includes multiple operations appearing in a specific order, it should be clearly understood that these processes may also include more or fewer operations, which can be executed sequentially or in parallel (e.g., using a parallel processor or a multi-threaded environment). Figure 5 As shown, the method includes the following steps: (S510~S530): S510 acquires an image of the environment surrounding the automated operating device captured by a structured light camera and generates a second counting identifier for the image.

[0149] like Figure 1 As shown, the second processing unit 130 can acquire images of the surrounding environment of the automated operating device from the structured light camera 120 via a bus and / or signal lines. After acquiring the images of the surrounding environment of the automated operating device, the second processing unit 130 can generate a second counting identifier for that frame image. For details, please refer to the foregoing description, which will not be repeated here.

[0150] S520, obtain the third counting identifier.

[0151] The third counting identifier is generated by the first processing unit based on the capturing signal, which is generated by the structured light camera based on the image acquisition action. For example... Figure 1 As shown, the second processing unit 130 can obtain the third counter identifier from the first processing unit 110 via a bus and / or signal lines. For details, please refer to the foregoing description, which will not be repeated here.

[0152] S530 determines whether the image acquisition time and the shooting signal generation time are synchronized based on the second and third counters.

[0153] The second processing unit can calculate the second difference between the third and second counting identifiers, and determine whether the absolute value of the second difference is greater than a second threshold, thereby determining whether the image acquisition time and the shooting signal generation time are synchronized. For details, please refer to the aforementioned descriptions, which will not be repeated here.

[0154] Figure 7 This is a schematic diagram of the structure of a time synchronization determination device for image acquisition in an automated operating device according to another embodiment of this application. This time synchronization determination device for image acquisition in an automated operating device can be applied to the second processing unit in the aforementioned automated operating device, such as... Figure 7 As shown, the time synchronization determination device for image acquisition in this automatic operation equipment may include: The second acquisition and generation module 710 is used to acquire images of the surrounding environment of the automated operating equipment captured by the structured light camera and generate a second counting identifier for the image; for details, please refer to the relevant description above, which will not be repeated here.

[0155] The second acquisition module 720 is used to acquire the third counting identifier; wherein the third counting identifier is generated by the first processing unit based on the shooting signal, and the shooting signal is generated by the structured light camera based on the image acquisition action; for details, please refer to the relevant description above, and will not be repeated here.

[0156] The second judgment module 730 is used to determine whether the image acquisition time and the shooting signal generation time are synchronized based on the second and third counting identifiers. For details, please refer to the aforementioned description, which will not be repeated here.

[0157] Obviously, the embodiments described above are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, those skilled in the art can make other variations or modifications without creative effort, and all such variations or modifications should fall within the scope of protection of this application.

Claims

1. A method for determining the time synchronization of image acquisition in an automatically operating device, characterized in that, The method includes: The first processing unit sends a current trigger signal to the structured light camera to trigger image acquisition, and generates a first count identifier and a first timestamp for the current trigger signal; The second processing unit acquires the current image captured by the structured light camera and generates a second counting identifier for the current image; the second processing unit caches a first counting identifier for multiple historical trigger signals and their corresponding first timestamps; The second processing unit determines whether the current image acquisition time and the current trigger signal generation time are synchronized based on the first counting identifier of the current trigger signal and the second counting identifier of the current image. If they are synchronized but the second counting identifier of the current image is inconsistent with the first counting identifier of the current trigger signal, then the matching historical trigger signal is searched from the plurality of historical trigger signals according to the second counting identifier of the current image, and the first timestamp of the matching historical trigger signal is used as the timestamp of the current image. The first processing unit controls the automatic operation device to work based on the current image and its timestamp.

2. The method according to claim 1, characterized in that, The second processing unit determines whether the current image acquisition time and the current trigger signal generation time are synchronized based on the first count identifier of the current trigger signal and the second count identifier of the current image, including: Calculate the first difference between the first count identifier of the current trigger signal and the second count identifier of the current image; If the absolute value of the first difference is greater than the first threshold, it is determined that the current image acquisition time and the current trigger signal generation time are out of sync.

3. The method according to claim 2, characterized in that, After determining that the current image acquisition time and the current trigger signal generation time are out of sync, the method further includes: The counts of the first count identifier of the current trigger signal and the second count identifier of the current image are reset.

4. The method according to claim 1, characterized in that, The method further includes: The second processing unit determines, based on the first counting identifier of the current trigger signal and the second counting identifier of the current image, that the current image acquisition time is synchronized with the current trigger signal generation time, and that the second counting identifier of the current image is consistent with the first counting identifier of the current trigger signal; Use the first timestamp of the current trigger signal as the timestamp of the current image.

5. The method according to any one of claims 1 to 4, characterized in that, The automated operation device further includes a data acquisition unit; the method further includes: The first processing unit acquires the data collected by the data acquisition unit and generates a second timestamp of the data; the data includes at least the current position of the automatic operation device; The second processing unit matches the current location and the current image based on the timestamp of the current image and the second timestamp; it then processes the matched current location and current image to generate a corresponding walking path. The first processing unit controls the automatic operation equipment to work according to the walking path.

6. The method according to claim 5, characterized in that, The second processing unit matches the current location and the current image based on the timestamp of the current image and the second timestamp, including: The second processing unit calculates the time difference between the second timestamp of the current location and the timestamp of the current image; If the time difference is less than a preset time threshold, then the current position is determined to match the current image.

7. A cleaning robot, characterized in that, The cleaning robot includes a first processing unit, a structured light camera, and a second processing unit. The structured light camera and the second processing unit are respectively connected to the first processing unit, and the structured light camera is connected to the second processing unit. The first processing unit is configured to: send a current trigger signal to the structured light camera, and generate a first counting identifier and a first timestamp for the current trigger signal; The structured light camera is used to: receive the current trigger signal and acquire current images of the environment surrounding the cleaning robot based on the current trigger signal; The second processing unit is used to: acquire the current image captured by the structured light camera and generate a second counting identifier for the current image; the second processing unit caches a first counting identifier for multiple historical trigger signals and their corresponding first timestamps; The second processing unit is further configured to: determine whether the current image acquisition time is synchronized with the current trigger signal generation time based on the first counting identifier of the current trigger signal and the second counting identifier of the current image; if they are synchronized but the second counting identifier of the current image is inconsistent with the first counting identifier of the current trigger signal, then search for a matching historical trigger signal from the plurality of historical trigger signals according to the second counting identifier of the current image, and use the first timestamp of the matching historical trigger signal as the timestamp of the current image; The first processing unit is also configured to: control the cleaning robot to work based on the current image and its timestamp.

8. A method for determining the time synchronization of image acquisition in an automatically operating device, characterized in that, The method includes: Acquire a current image of the environment surrounding the automatically operating equipment captured by a structured light camera, and generate a second counting identifier for the current image; Obtain a first count identifier and a first timestamp; wherein the first count identifier and the first timestamp are generated by the first processing unit based on the current trigger signal, the current trigger signal being generated by the first processing unit to enable the structured light camera to acquire an image; Based on the first and second counting identifiers, it is determined whether the current image acquisition time and the current trigger signal generation time are synchronized. If they are synchronized but the second counting identifier of the current image is inconsistent with the first counting identifier of the current trigger signal, then according to the second counting identifier of the current image, a matching historical trigger signal is searched from multiple historical trigger signals cached locally, and the first timestamp of the matching historical trigger signal is used as the timestamp of the current image. The current image and its timestamp are used by the first processing unit to control the automatic operation device.

9. A time synchronization judgment device for image acquisition in an automatic operating device, characterized in that, The device includes: The first acquisition and generation module is used to acquire the current image of the environment surrounding the automatically operating equipment captured by the structured light camera, and generate a second counting identifier for the current image; The first acquisition module is used to acquire a first count identifier and a first timestamp; wherein the first count identifier and the first timestamp are generated by the first processing unit based on a current trigger signal, and the current trigger signal is generated by the first processing unit to enable the structured light camera to acquire an image; The first judgment module is used to determine whether the current image acquisition time and the current trigger signal generation time are synchronized based on the first counting identifier and the second counting identifier. If they are synchronized but the second counting identifier of the current image is inconsistent with the first counting identifier of the current trigger signal, then according to the second counting identifier of the current image, a matching historical trigger signal is searched from multiple historical trigger signals cached locally, and the first timestamp of the matching historical trigger signal is used as the timestamp of the current image. The current image and its timestamp are used by the first processing unit to control the operation of the automatic operation device.

10. A method for determining the time synchronization of image acquisition in an automatically operating device, characterized in that, The method includes: The structured light camera acquires a current image of the environment surrounding the automated equipment, generates a current shooting signal and sends it to the first processing unit, and sends the current image to the second processing unit. The first processing unit receives the current shooting signal and generates a third counting identifier and a third timestamp for the current shooting signal; The second processing unit receives the current image and generates a second counting identifier for the current image; the second processing unit caches a third counting identifier for multiple historical shooting signals and their corresponding third timestamps; The second processing unit determines whether the current image acquisition time and the current image generation time are synchronized based on the third counting identifier of the current shooting signal and the second counting identifier of the current image. If they are synchronized but the second counting identifier of the current image is inconsistent with the third counting identifier of the current shooting signal, then it searches for a matching historical shooting signal from the plurality of historical shooting signals according to the second counting identifier of the current image, and uses the third timestamp of the matching historical shooting signal as the timestamp of the current image. The first processing unit controls the automatic operation device to work based on the current image and its timestamp.

11. The method according to claim 10, characterized in that, The second processing unit determines whether the current image acquisition time and the current image generation time are synchronized based on the third counter identifier of the current captured signal and the second counter identifier of the current image, including: Calculate the second difference between the third count identifier of the current captured signal and the second count identifier of the current image; If the absolute value of the second difference is greater than the second threshold, it is determined that the current image acquisition time is out of sync with the current shooting signal generation time.

12. The method according to claim 11, characterized in that, After determining that the current image acquisition time and the current shooting signal generation time are out of sync, the method further includes: The counts of the third counter identifier of the current captured signal and the second counter identifier of the current image are reset.

13. The method according to claim 10, characterized in that, The method further includes: The second processing unit determines, based on the third counting identifier of the current shooting signal and the second counting identifier of the current image, that the current image acquisition time is synchronized with the current shooting signal generation time, and that the second counting identifier of the current image is consistent with the third counting identifier of the current shooting signal; Use the third timestamp of the current captured signal as the timestamp of the current image.

14. The method according to any one of claims 10 to 13, characterized in that, The automated operation device further includes a data acquisition unit; the method further includes: The first processing unit acquires the data collected by the data acquisition unit and generates a second timestamp of the data; the data includes at least the current position of the automatic operation device; The second processing unit matches the current location and the current image based on the timestamp of the current image and the second timestamp; it then processes the matched current location and current image to generate a corresponding walking path. The first processing unit controls the automatic operation equipment to work according to the walking path.

15. The method according to claim 14, characterized in that, The second processing unit matches the current location and the current image based on the timestamp of the current image and the second timestamp, including: The second processing unit calculates the time difference between the second timestamp of the current location and the timestamp of the current image; If the time difference is less than a preset time threshold, then the current position is determined to match the current image.

16. A cleaning robot, characterized in that, The cleaning robot includes a first processing unit, a structured light camera, and a second processing unit. The structured light camera and the second processing unit are respectively connected to the first processing unit, and the structured light camera is connected to the second processing unit. The structured light camera is used to: acquire current images of the environment surrounding the cleaning robot, generate a current shooting signal and send it to the first processing unit, and send the current image to the second processing unit; The first processing unit is configured to: receive the current shooting signal, and generate a third counting identifier and a third timestamp of the current shooting signal; The second processing unit is configured to: receive the current image and generate a second counting identifier for the current image; the second processing unit caches a third counting identifier for multiple historical shooting signals and their corresponding third timestamps; The second processing unit is further configured to: determine whether the current image acquisition time is synchronized with the current image generation time based on the third counting identifier of the current shooting signal and the second counting identifier of the current image; if they are synchronized but the second counting identifier of the current image is inconsistent with the third counting identifier of the current shooting signal, then according to the second counting identifier of the current image, search for a matching historical shooting signal from the plurality of historical shooting signals, and use the third timestamp of the matching historical shooting signal as the timestamp of the current image; The first processing unit is also configured to: control the cleaning robot to work based on the current image and its timestamp.

17. A method for determining the time synchronization of image acquisition in an automatically operating device, characterized in that, The method includes: Acquire a current image of the environment surrounding the automatically operating equipment captured by a structured light camera, and generate a second counting identifier for the current image; Obtain a third counting identifier and a third timestamp; wherein the third counting identifier and the third timestamp are generated by the first processing unit based on the current shooting signal, and the current shooting signal is generated by the structured light camera based on the image acquisition action; Based on the second and third counting identifiers, it is determined whether the current image acquisition time and the current shooting signal generation time are synchronized. If they are synchronized but the second counting identifier of the current image is inconsistent with the third counting identifier of the current shooting signal, then according to the second counting identifier of the current image, a matching historical shooting signal is searched from multiple locally cached historical shooting signals, and the third timestamp of the matching historical shooting signal is used as the timestamp of the current image. The current image and its timestamp are used by the first processing unit to control the operation of the automatic operation device.

18. A time synchronization judgment device for image acquisition in an automatic operating device, characterized in that, The device includes: The second acquisition and generation module is used to acquire the current image of the environment surrounding the automatically operating equipment captured by the structured light camera, and generate a second counting identifier for the current image; The second acquisition module is used to acquire a third counting identifier and a third timestamp; wherein the third counting identifier and the third timestamp are generated by the first processing unit based on the current shooting signal, and the current shooting signal is generated by the structured light camera based on the image acquisition action; The second judgment module is used to determine whether the current image acquisition time and the current shooting signal generation time are synchronized based on the second and third counting identifiers. If they are synchronized but the second counting identifier of the current image is inconsistent with the third counting identifier of the current shooting signal, then according to the second counting identifier of the current image, a matching historical shooting signal is searched from multiple locally cached historical shooting signals, and the third timestamp of the matching historical shooting signal is used as the timestamp of the current image. The current image and its timestamp are used by the first processing unit to control the operation of the automatic operation device.

Images