Carrying equipment slip detection method and device and carrying equipment

Abstract

The invention relates to the technical field of warehouse logistics, and discloses a carrying equipment slip detection method and device and carrying equipment, the method comprises the following steps: in the process that the carrying equipment moves along a driving path, acquiring first detection data acquired by an image acquisition device and second detection data acquired by an odometer; wherein the first detection data comprises a plurality of initial image frames; determining a target image frame meeting a preset condition in the plurality of initial image frames, and determining a first position of the first feature point in the first image frame and a second position of the first feature point in a second image frame adjacent to the first image frame; determining a first detection distance corresponding to the image acquisition device based on the first position and the second position, and determining a second detection distance corresponding to the odometer based on the second detection data; and based on the first detection distance and the second detection distance, the slipping condition of the carrying equipment is determined. On the premise that the deployment cost is not increased, the slip detection precision is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of warehouse logistics, in particular to a skid detection method and device for a carrying device and the carrying device. BACKGROUND

[0002] A four-way shuttle vehicle (hereinafter referred to as a four-way vehicle) may slip due to wet and slippery running tracks, worn wheel systems, or uneven loads during operation. The pose of the four-way vehicle after slipping changes, and if the slipping of the four-way vehicle is not determined in time to calibrate the pose of the four-way vehicle, the accuracy of subsequent positioning of the four-way vehicle will be affected.

[0003] Generally, the four-way vehicle can determine whether the four-way vehicle slips based on a wheeled odometer and an inertial measurement unit. However, the position determined by the inertial measurement unit based on inertia has a large error, which leads to a large deviation in subsequent slip determination. Other slip detection methods may require corresponding modification of the track of the four-way vehicle, which has high deployment costs and poor adaptability. Therefore, there is an urgent need for a slip detection method that can achieve high precision, low cost, and high reliability of the four-way vehicle in complex environments. SUMMARY

[0004] To solve the above problems, the embodiments of the present application provide a carrying device slip detection method, device and carrying device, which can improve the slip detection accuracy of the four-way vehicle without increasing the deployment cost. Specifically, the embodiments of the present application disclose the following technical solutions:

[0005] The first aspect of the embodiments of the present application provides a carrying device slip detection method applied to a carrying device, wherein the carrying device is provided with an image acquisition device and an odometer. The method comprises: acquiring first detection data collected by the image acquisition device and second detection data collected by the odometer during movement of the carrying device along a driving path; wherein the first detection data comprises a plurality of initial image frames; determining a target image frame that satisfies a preset condition in the plurality of initial image frames, and determining a first position of a first feature point in a first image frame and a second position of the first feature point in a second image frame adjacent to the first image frame; wherein the target image frame comprises the first image frame and the second image frame; determining a first detection distance corresponding to the image acquisition device based on the first position and the second position, and determining a second detection distance corresponding to the odometer based on the second detection data; and determining a slip condition of the carrying device based on the first detection distance and the second detection distance.

[0006] In some embodiments, the determining the target image frame satisfying the preset condition from the plurality of initial image frames comprises: determining a brightness of each initial image frame and a number of feature points on each initial image frame; wherein the travel path of the conveying device comprises a track, and the feature points are used to represent identification information of the track; and determining, as the target image frame satisfying the preset condition, an initial image frame from the plurality of initial image frames, the initial image frame having a brightness within a preset brightness value range and / or a number of feature points greater than a first preset number threshold.

[0007] In some embodiments, before the determining the first position of the first feature point in the first image frame and the second position of the first feature point in the second image frame adjacent to the first image frame, the method further comprises: determining a plurality of first initial feature points in the first image frame and a plurality of second initial feature points in the second image frame; and determining at least one target feature point based on the first initial feature points and the second initial feature points; wherein the at least one target feature point is located in both the first image frame and the second image frame, and the at least one target feature point comprises the first feature point.

[0008] In some embodiments, the determining the at least one target feature point based on the first initial feature points and the second initial feature points comprises: determining a plurality of first candidate feature points located in both the first image frame and the second image frame based on the first initial feature points and the second initial feature points; determining a first candidate position of each first candidate feature point in the first image frame and a second candidate position of each first candidate feature point in the second image frame; determining a displacement difference value between the first candidate position and the second candidate position corresponding to each first candidate feature point; and determining the at least one target feature point from the plurality of first candidate feature points based on the displacement difference values.

[0009] In some embodiments, the determining the at least one target feature point from the plurality of first candidate feature points based on the displacement difference values comprises: determining, as the target feature point, a first candidate feature point from the plurality of first candidate feature points, the first candidate feature point having a displacement difference value reaching a preset displacement difference threshold.

[0010] In some embodiments, the method further comprises: determining a number of first candidate feature points and a number of second candidate feature points; wherein the second candidate feature points are first candidate feature points from the plurality of first candidate feature points, the first candidate feature points having displacement difference values not reaching the preset displacement difference threshold; and determining an abnormal feature point proportion based on the number of first candidate feature points and the number of second candidate feature points.

[0011] In some embodiments, the determining the skidding of the carrying device based on the first detection distance and the second detection distance comprises: determining a first distance difference between the first detection distance and the second detection distance when the proportion of the abnormal feature points is less than the preset proportion threshold; determining that the carrying device skids when the first distance difference exceeds a target distance threshold; or determining that the carrying device skids when the first distance difference exceeds the target distance threshold and a duration for which the first distance difference exceeds the target distance threshold exceeds a target time threshold.

[0012] In some embodiments, the carrying device moves on the track by the walking wheels, and the method further comprises: determining a friction coefficient between the walking wheels of the carrying device and the track, and a carrying state of the carrying device; wherein the carrying state comprises an empty state and a loaded state; and determining the target distance threshold and the target time threshold based on the friction coefficient and / or the carrying state.

[0013] In some embodiments, the method further comprises: discarding the target image frame and re-acquiring the detection information of the image acquisition device when the proportion of the abnormal feature points is greater than or equal to the preset proportion threshold.

[0014] In some embodiments, the method further comprises, before the determining the first initial feature points in the first image frame and the second initial feature points in the second image frame: performing block processing on the plurality of initial feature points of the target image frame to divide the target image frame into a plurality of grid units; and performing screening processing on the grid units including the plurality of initial feature points to make the number of initial feature points in the grid units less than or equal to a second preset number threshold.

[0015] In some embodiments, the method further comprises: determining the number of feature points in a third image frame of the plurality of initial image frames; determining whether the first reference point is included in the third image frame when the number of feature points in the third image frame is less than a third preset number threshold; determining the third detection distance corresponding to the image acquisition device based on the first reference point and the second reference point when the third image frame includes the first reference point and the fourth image frame including the second reference point is acquired by the image acquisition device, and determining the fourth detection distance corresponding to the odometer; determining a second distance difference between the third detection distance and the fourth detection distance, and determining that the carrying device skids when the second distance difference exceeds the target distance threshold.

[0016] In some embodiments, the carrying device moves on the track by the walking wheels, and the track comprises a plurality of sub-tracks, and the reference points are used to represent the connection identifiers between two sub-tracks; and the detection distances corresponding to two adjacent reference points are the lengths of the sub-tracks between the two adjacent reference points.

[0017] In some embodiments, the carrying device includes first walking wheels and second walking wheels, and the image acquisition device includes a first optical flow sensor arranged between the two first walking wheels and a second optical flow sensor arranged between the two second walking wheels. The first detection data collected by the image acquisition device during the movement of the carrying device along the driving path is obtained by: obtaining detection information corresponding to the first track collected by the first optical flow sensor during the movement of the carrying device on the first track by the first walking wheels; or obtaining detection information corresponding to the second track collected by the second optical flow sensor during the movement of the carrying device on the second track by the second walking wheels.

[0018] In some embodiments, after determining the skidding condition of the carrying device, the method further includes: when it is determined that the carrying device is in a skidding condition, controlling the output device to output alarm information, and adjusting the position of the carrying device based on the first detection distance.

[0019] The second aspect of the embodiments of the present application provides a carrying device skidding detection device, which includes an acquisition module, a first determination module, a second determination module and a third determination module. The acquisition module is configured to: during the movement of the carrying device along the driving path, acquire first detection data collected by an image acquisition device of the carrying device and second detection data collected by an odometer of the carrying device; wherein the first detection data includes a plurality of initial image frames. The first determination module is configured to: determine a target image frame that meets a preset condition in the plurality of initial image frames, and determine a first position of a first feature point in a first image frame and a second position of the first feature point in a second image frame adjacent to the first image frame; wherein the target image frame includes the first image frame and the second image frame. The second determination module is configured to: determine a first detection distance corresponding to the image acquisition device based on the first position and the second position, and determine a second detection distance corresponding to the odometer based on the second detection data. The third determination module is configured to: determine a skidding condition of the carrying device based on the first detection distance and the second detection distance.

[0020] The third aspect of the embodiment of the present application provides a carrying device, comprising an image acquisition device, an odometer and a control device. The image acquisition device is configured to acquire first detection data in the process that the carrying device moves along a driving path, wherein the first detection data comprises a plurality of initial image frames. The odometer is configured to acquire second detection data in the process that the carrying device moves along the driving path. The control device is configured to acquire the first detection data and the second detection data, determine a target image frame that satisfies a preset condition in the plurality of initial image frames, and determine a first position of a first feature point in a first image frame and a second position of the first feature point in a second image frame adjacent to the first image frame, wherein the target image frame comprises the first image frame and the second image frame; determine a first detection distance corresponding to the image acquisition device based on the first position and the second position, and determine a second detection distance corresponding to the odometer based on the second detection data; and determine a skid condition of the carrying device based on the first detection distance and the second detection distance.

[0021] The fourth aspect of the embodiment of the present application provides an electronic device, comprising a processor and a memory, wherein the memory is used to store computer executable instructions; and the processor is used to read the instructions from the memory and execute the instructions to implement the carrying device skid detection method of the first aspect.

[0022] The fifth aspect of the embodiment of the present application provides a computer readable storage medium, wherein the storage medium stores computer program instructions, and when a computer reads the instructions, the carrying device skid detection method of the first aspect is executed.

[0023] The sixth aspect of the embodiment of the present application provides a computer program product, comprising a computer program stored on a non-transitory computer readable storage medium, wherein the computer program comprises program instructions, and when the program instructions are executed by a computer, the computer executes the carrying device skid detection method of the first aspect.

[0024] The seventh aspect of the embodiment of the present application provides a computer program, when the computer program is executed by a processor, the carrying device skid detection method of the first aspect can be implemented.

[0025] The carrying equipment slip detection method, device and carrying equipment provided by the embodiment of the application detect the slip of the carrying equipment (such as a four-way vehicle) through an image acquisition device (such as an optical flow sensor) and an odometer. The optical flow sensor can acquire image frames corresponding to the identification information (such as the texture of the track) on the track during the movement of the four-way vehicle. The control device determines the positions of the same feature points in adjacent image frames, thereby determining the first detection distance (such as the actual displacement) of the carrying equipment. Then, the second detection distance (such as the theoretical displacement) of the carrying equipment is determined through the detection data of the odometer. Finally, the actual displacement corresponding to the optical flow sensor and the theoretical displacement corresponding to the odometer are combined to determine whether the carrying equipment slips. The embodiment of the application realizes the detection of whether the carrying equipment slips through the cooperation of the optical flow sensor and the odometer. Since the detection of the optical flow sensor depends on the texture information of the track itself, the track does not need to be modified and is not affected by the environment, and the practicality is high and the detection precision is high. Therefore, the embodiment of the application can realize high-precision, low-cost and high-reliability slip detection of the four-way vehicle in a complex environment. BRIEF DESCRIPTION OF DRAWINGS

[0026] In order to more clearly illustrate the technical solutions of the embodiments of the application, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0027] Figure 1 A schematic diagram of a warehouse system provided by the embodiment of the application;

[0028] Figure 2 A schematic diagram of a carrying equipment provided by the embodiment of the application;

[0029] Figure 3 A schematic diagram of a carrying equipment provided by the embodiment of the application;

[0030] Figure 4 A schematic diagram of another carrying equipment provided by the embodiment of the application;

[0031] Figure 5 A schematic diagram of a carrying equipment slip detection method provided by the embodiment of the application;

[0032] Figure 6 A schematic diagram of another carrying equipment slip detection method provided by the embodiment of the application;

[0033] Figure 7 A schematic diagram of another carrying equipment slip detection method provided by the embodiment of the application;

[0034] Figure 8A schematic diagram of a moving process of a carrying device provided by an embodiment of the present application is shown in FIG. 1.

[0035] Figure 9 A schematic diagram of a skid detection process of a carrying device provided by an embodiment of the present application is shown in FIG. 2.

[0036] Figure 10 A schematic diagram of another skid detection process of a carrying device provided by an embodiment of the present application is shown in FIG. 3.

[0037] Figure 11 A schematic diagram of a skid detection device of a carrying device provided by an embodiment of the present application is shown in FIG. 4.

[0038] Figure 12 A schematic diagram of an electronic device provided by an embodiment of the present application is shown in FIG. 5. DETAILED DESCRIPTION

[0039] In order to make the personnel in the technical field better understand the technical solutions in the embodiments of the present application, and make the above-mentioned purposes, characteristics and advantages of the embodiments of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application are further described in detail below with reference to the drawings.

[0040] Figure 1 A schematic diagram of a warehouse system provided by an embodiment of the present application is shown in FIG. 6. Figure 1 As shown, the warehouse system includes a carrier 10 and a carrying device 20.

[0041] In some examples, the carrier 10 can be a shelf with multiple layers of storage locations, and each layer of the carrier 10 can be provided with a lane 300, which can extend along a first direction. For example, each layer of the carrier 10 can be provided with multiple lanes 300, which can be spaced apart along a second direction. At least one side of each storage location has a lane 300, so that the article P taken out from any one storage location can be moved out from the lane 300 on one side.

[0042] It should be noted that the article P includes but is not limited to goods, original boxes or packages containing goods, containers containing goods or original boxes, etc., wherein the container can include but is not limited to a box, a pallet, etc.

[0043] In some examples, each layer of the carrier 10 is also provided with a goods aisle 400, and the storage location is arranged on the goods aisle 400. The goods aisle 400 can extend along the second direction of the shelf to communicate with the multiple lanes 300 arranged along the first direction. It should be noted that each layer of the shelf can be spaced apart along the first direction to have multiple goods aisles 400, and multiple storage locations are spaced apart on each goods aisle 400, so that the storage density of each layer of the shelf can be improved.

[0044] As shown in FIG. 7, Figure 1As shown, one or two storage locations can be provided between two adjacent aisles 300. Taking two storage locations as an example, the items P on one of the storage locations can be removed from the left aisle 300, and the items on the other storage location can be removed from the right aisle 300.

[0045] Figure 2 A schematic view of a carrying device provided in an embodiment of the present application is shown. Figure 3 A schematic view of a carrier cooperating with a carrying device provided in an embodiment of the present application is shown. It should be noted that, Figure 2 As shown, the view can be a bottom view of the carrying device 20.

[0046] Exemplarily, as Figure 2 shown, the carrying device 20 can include first traveling wheels 21, second traveling wheels 22, and a vehicle body 23. The vehicle body 23 includes a first side, a second side, a third side, and a fourth side, the first side and the second side are oppositely arranged, and the third side and the fourth side are oppositely arranged. The first traveling wheels 21 are arranged on the first side and the second side of the vehicle body 23, and the second traveling wheels 22 are arranged on the third side and the fourth side of the vehicle body 23. The vehicle body 23 is driven to travel by the first traveling wheels 21 or the second traveling wheels.

[0047] In some examples, the carrying device 20 can also be referred to as a four-way vehicle robot (hereinafter referred to as a four-way vehicle). The four-way vehicle can include two groups of first traveling wheels 21, one group of first traveling wheels 21 is arranged on the first side of the vehicle body 23, and the other group of first traveling wheels 21 is arranged on the second side of the vehicle body 23. The number of wheels in each group of first traveling wheels 21 can be one or more (such as Figure 2 two). Similarly, the four-way vehicle robot can also include two groups of second traveling wheels 22, one group of second traveling wheels 22 is arranged on the third side of the vehicle body 23, and the other group of second traveling wheels 22 is arranged on the fourth side of the vehicle body 23. The number of wheels in each group of second traveling wheels 22 can be one or more (such as Figure 2 two), which is not limited in the embodiments of the present application.

[0048] Exemplarily, in order to enable the carrying device 20 to stably move on the aisles 300 and the shelves 400, the carrier 10 further includes a first track 100 and a second track 200.

[0049] As Figure 3 shown, the first track 100 is arranged on one of the aisles 300 and the shelves 400 of the carrier 10, and the second track 200 is arranged on the other one of the aisles 300 and the shelves 400. The carrying device 20 can move on each layer of the carrier 10 along the first track 100 and the second track 200. For example, when the first track 100 is arranged on the aisle 300, the second track can be arranged on the shelf 400.

[0050] For example, the first tracks 100 can be arranged in the aisles 300 for the carrying devices 20 to travel in the aisles 300. The second tracks 200 can be arranged in the lanes 400 for the carrying devices 20 to travel in the lanes 400. The present application does not limit the arrangement of the first tracks 100 and the second tracks 200. The following embodiments are exemplarily described with the first tracks 100 arranged in the aisles 300 and the second tracks 200 arranged in the lanes 400.

[0051] In some examples, the first traveling wheels 21 of the carrying devices 20 can roll along the first tracks 100 to move the carrying devices 20 along the first tracks 100, wherein the first tracks 100 can extend in the first direction. The second traveling wheels 22 of the carrying devices 20 can roll along the second tracks 200 to move the carrying devices 20 along the second tracks 200, wherein the second tracks 200 can extend in the second direction.

[0052] Exemplarily, each aisle 300 can be provided with two first tracks 100, which are arranged in the width direction (e.g. the second direction) of the aisle 300 and used for the first traveling wheels 21 arranged on the first side and the second side of the carrying devices 20. For example, the first traveling wheels 21 on the opposite sides (i.e. the first side and the second side) of the carrying devices 20 in the second direction can be supported on the corresponding first tracks 100 and travel along the first tracks 100 in the first direction. It should be noted that the following embodiments can refer to the two first tracks 100 on one aisle 300 cooperating with the same carrying device 20 as a first track group 101 (as shown in Figure 1

[0053] In some examples, the first tracks 100 can be arranged on the frames of each layer of the carriers 10. For example, the first tracks 100 can be detachably arranged on the frames of each layer of the carriers 10 to facilitate the assembly stability of the first tracks 100 on the carriers and facilitate the disassembly of the first tracks 100. In other examples, the first tracks 100 can directly serve as the frames of each layer of the carriers, i.e. one first track group forms one aisle 300 of each layer of the carriers.

[0054] Exemplarily, the carrying devices 20 can travel on the second tracks 200 by the second traveling wheels 22 to reach the target storage locations of the lanes 400 and interface with the target storage locations to take or return the articles P.

[0055] In some examples, as Figure 3 ​As shown, the carrier 10 comprises a support frame 220, and a support surface 220a of the support frame 220 is used to support the container, i.e., the support surface 220a of the support frame 220 serves as a storage location, and the container is placed on the support surface 220a of the support frame 220. The support surface 220a of the support frame 220 is located above the running surface of the second track 200, so as to form a running space (not shown in the figure) between the support surface 220a and the running surface, which is used for the smooth passing of the carrying device 20 below the storage location. Figure 3

[0056] Exemplarily, the vehicle body 23 can comprise a bracket and a base, the first walking wheel 21 is arranged on the bracket or the base, and the second walking wheel 22 is arranged on the base, and the bracket can be lifted up and down relative to the base of the carrying device 20. When the carrying device 20 in an empty state reaches below the target storage location along the second track 200, the bracket can be controlled to be lifted up until the container on the target storage location is lifted up. After the bracket lifts up the container on the target storage location, the carrying device 20 carrying the container continues to run in the storage channel 400 along the second track 200 until it reaches the nearby aisle 300, and the container is moved out of the carrier from the aisle 300, thereby completing the picking process. Similarly, when the carrying device 20 carrying the container reaches the target storage location along the second track 200, the bracket can be controlled to be lowered so that the container on the bracket is lowered to be in contact with the support surface 220a of the support frame 220. After the container on the bracket is lowered to be in contact with the support surface 220a of the support frame 220, the bracket continues to be lowered, so that the bracket is separated from the container, and the returning process is completed.

[0057] Exemplarily, two second tracks 200 are arranged on each storage channel 400, and the two second tracks 200 are arranged in a width direction (e.g., a first direction) of the storage channel 400, and the second walking wheel 22 on the opposite sides of the carrying device 20 can be supported on the corresponding second track 200 and walk along the second track 200 in a second direction. For the convenience of description, two second tracks 200 on one storage channel 400 that cooperate with the same carrying device 20 are referred to as a second track group 201 (as shown). Figure 1

[0058] In some examples, the second track 200 can be arranged on the frame of each layer of the carrier. For example, the second track 200 can be detachably arranged on the frame of each layer of the carrier, so as to facilitate the assembly stability of the second track 200 on the carrier and facilitate the disassembly of the second track 200. In another example, the second track 200 can directly serve as the frame of each layer of the carrier, i.e., one second track group 201 forms one storage channel 400 of each layer of the carrier.

[0059] For example, the first track 100 and the second track 200 are arranged in a cross manner, i.e., the first track 100 and the second track 200 have an overlapping support region​​Figure 3 The overlapping support area (not shown in the figure) is used for the switching of the transport device 20 between the aisle 300 and the goods aisle 400.

[0060] In some examples, the four-way vehicle can run on the track (such as the first track 100 or the second track 200) in the warehouse, and slippage can occur. For example, when the track or the ground is wet, greasy, or the like, which can reduce the friction coefficient between the walking wheel and the contact surface, the wheel driving force exceeds the maximum static friction that the ground can provide, and the wheel slips. For another example, the walking wheel can be worn after long-term use, which can cause the walking wheel to be not flexible in rotation or the rolling resistance of different walking wheels to be inconsistent, thereby causing slippage. For another example, when the four-way vehicle is loaded with goods, the center of gravity is offset, and slippage can occur. When slippage occurs, if it cannot be timely, it can cause a large deviation in the positioning of the four-way vehicle, thereby affecting the execution of the task of the four-way vehicle. Therefore, when the four-way vehicle slips, it needs to be timely determined and adjusted.

[0061] Generally, the four-way vehicle can determine whether it slips by the detection data of the wheel odometer and the inertial measurement unit. When the theoretical distance calculated by the wheel odometer and the actual distance calculated by the inertial measurement unit are obviously deviated, it can be determined that the wheel of the four-way vehicle slips. However, since the inertial measurement unit determines the position of the four-way vehicle based on inertia, when slippage occurs, the real position determined by the inertial measurement unit can be inaccurate, thereby causing a deviation in the judgment of whether the transport device slips. On the other hand, the four-way vehicle can also determine whether the transport device slips based on the wheel odometer and the photoelectric sensor. The photoelectric sensor mainly determines the position by detecting the detection hole on the track, and thus when the photoelectric sensor is detected, the track needs to be punched or the like, that is, the photoelectric sensor depends on the hardware modification, and has the problems of high hardware cost and poor adaptability.

[0062] To solve the above problems, the embodiment of the present application provides a transport device, the transport device is provided with an optical flow sensor, the optical flow sensor collects the identification information (such as texture information) on the track, and determines the actual distance of the transport device based on the pixel displacement of the feature points corresponding to the identification information in the continuous image frames, and determines whether the transport device slips based on the distance difference between the actual distance determined by the optical flow sensor and the theoretical distance determined by the odometer. The embodiment of the present application improves the accuracy of the slippage detection of the transport device, has small detection error, and does not need to modify the track, thereby reducing the deployment cost of the warehouse.

[0063] Figure 4 A schematic view of another transport device provided by the embodiment of the present application.

[0064] In some embodiments, the image acquisition device is arranged on the carrying device 20. The number of image acquisition devices can be multiple, for example, the number of image acquisition devices can be two. The image acquisition device can be an optical flow sensor or other device capable of image acquisition, and the number of image acquisition devices is not limited in the embodiments of the present application. The following embodiments take the optical flow sensor as an example for illustrative description.

[0065] As shown in the example, Figure 4 When the number of optical flow sensors 24 is two, the two optical flow sensors 24 can include a first optical flow sensor 241 and a second optical flow sensor 242. The optical flow sensor 24 can be arranged at the bottom of the carrying device 20, and used to acquire the identification information on the track (such as the first track or the second track).

[0066] In some examples, the track has identification information, which can be texture information on the track. The texture information can be self-provided by the track, such as scratches, printed line spots and texture on the surface of the track, or can be a connection mark between two adjacent tracks. The number of optical flow sensors is not limited in the embodiments of the present application. The following examples take the optical flow sensor to acquire the texture information on the track as an example for illustrative description.

[0067] In some examples, the first optical flow sensor 241 can be arranged between the two first walking wheels 21, and the second optical flow sensor 242 can be arranged between the two second walking wheels 22. For example, as shown in the example, Figure 4 The first optical flow sensor 241 can be arranged between the two first walking wheels 21 on the first side of the carrying device 20, and the second optical flow sensor 242 can be arranged between the two second walking wheels 22 on the third side of the carrying device 20.

[0068] It should be noted that the first optical flow sensor 241 can also be arranged between the two first walking wheels 21 on the second side of the carrying device 20, and the second optical flow sensor 242 can also be arranged between the two second walking wheels 22 on the fourth side of the carrying device 20. The number of optical flow sensors 24 is not limited in the embodiments of the present application. For example, the carrying device 20 can also have more optical flow sensors 24, for example, when the number of optical flow sensors 24 is four, one optical flow sensor can be arranged on each side of the carrying device 30. The number of optical flow sensors 24 is not limited in the embodiments of the present application. In order to further reduce the hardware deployment cost, two optical flow sensors are arranged in the embodiments of the present application. The following examples take the carrying device with two optical flow sensors as an example for illustrative description.

[0069] In some examples, in order to ensure the collection accuracy, the optical flow sensor 24 can be arranged at a height less than or equal to a preset height position from the track surface. The preset height can be set according to requirements. For example, the preset height can be 5 cm. The optical flow sensor 24 can adopt a model with a high frame rate (e.g., greater than or equal to 60 fps) and a high resolution (e.g., greater than or equal to 500 dpi) to improve the collection accuracy and resolution.

[0070] For example, the optical flow sensor 24 is composed of an optical lens and a photosensitive element. The optical lens is used to focus the texture light of the track surface onto the photosensitive element. The photosensitive element receives the focused light and converts the light signal into an electrical signal (i.e., an image frame). The photosensitive surface, i.e., the physical surface of the photosensitive element, can be used to determine the orientation and angle of the shooting area of the optical flow sensor 24. In the four-way yard scenario, the photosensitive surface axis can be perpendicular to the extension direction of the track, i.e., the photosensitive surface axis is perpendicular to the length direction of the track, thereby being used to collect the displacement of the track surface texture.

[0071] For example, the handling device 20 further includes an odometer (e.g., a wheeled odometer). The odometer can collect the data of the walking wheel (e.g., the first walking wheel 21 or the second walking wheel 22) of the handling device 20 in real time and transmit the data to the control device (e.g., a central controller) of the handling device 20 through a controller area network (CAN).

[0072] In some examples, the optical flow sensor can also be coupled to the control device through the CAN bus to send the collected image frames containing the track texture to the control device. The control device receives the detection data sent by the optical flow sensor and the detection data of the odometer, and further determines whether the handling device slips.

[0073] The handling device slip detection method provided by the embodiment of the present application will be described below with reference to the accompanying drawings.

[0074] Figure 5 A schematic diagram of a handling device slip detection method provided by an embodiment of the present application is shown in FIG. 5. As shown in FIG. 5, the method includes the following steps 510 to 540. Figure 5

[0075] Step 510, during the movement of the handling device along the driving path, the first detection data collected by the image collection device and the second detection data collected by the odometer are obtained.

[0076] For example, as shown in FIG. 6, the handling device 20 further includes an odometer 40. The odometer 40 can collect the data of the walking wheel (e.g., the first walking wheel 21 or the second walking wheel 22) of the handling device 20 in real time and transmit the data to the control device (e.g., a central controller) of the handling device 20 through a controller area network (CAN). Figure 4 ​As shown, when the carrying device is a four-way vehicle, the travel path of the carrying device can be a track, such as the first track 100 or the second track 200. Among them, the first track 100 can be referred to as a main track, and the second track 200 can be referred to as a sub-track. The first walking wheel 21 moving on the first track 100 can be referred to as a main wheel, and the second walking wheel 22 moving on the second track 200 can be referred to as a sub-wheel.

[0077] In some examples, the main wheel rolls along the main track to drive the carrying device to move along the main track; and the sub-wheel rolls along the sub-track to drive the carrying device to move along the sub-track. During the travel of the carrying device along the first track (or the second track), the image acquisition device on the carrying device can acquire texture information on the track in real time to obtain first detection data. The first detection data includes a plurality of initial image frames, and the initial image frames include feature points corresponding to the texture information.

[0078] For example, the plurality of initial image frames can be a plurality of image frames continuously acquired by the optical flow sensor, such as two image frames, or a larger number of image frames, which are not limited in the embodiments of the present application.

[0079] In some embodiments, the carrying device includes first walking wheels and second walking wheels, and the image acquisition device includes a first optical flow sensor arranged between the two first walking wheels, and a second optical flow sensor arranged between the two second walking wheels. Step 510 includes: acquiring detection information corresponding to the first track acquired by the first optical flow sensor during the movement of the carrying device on the first track by the first walking wheels; or acquiring detection information corresponding to the second track acquired by the second optical flow sensor during the movement of the carrying device on the second track by the second walking wheels.

[0080] For example, the first optical flow sensor can acquire detection information corresponding to the first track (which can be referred to as main detection information) during the movement of the main wheel along the main track, and send the main detection information to the control device. The second optical flow sensor can acquire detection information corresponding to the second track (which can be referred to as sub-detection information) during the movement of the sub-wheel along the sub-track, and send the main detection information to the control device. The first detection information includes the main detection information or the main detection information.

[0081] In some examples, during the movement of the carrying device 20 along the track, the first and second optical flow sensors can be in an open state, and the first and second optical flow sensors can each collect detection data of the corresponding track. For example, during the movement of the first walking wheel along the first track, the first optical flow sensor can collect texture information of the first track, and the second optical flow sensor cannot collect texture information on the first track, so that the control device can perform subsequent processing based on the detection information detected by the first optical flow sensor. Similarly, during the movement of the second walking wheel along the second track, the first optical flow sensor cannot collect texture information of the first track, and the second optical flow sensor can collect texture information on the second track, so that the control device can perform subsequent processing based on the detection information detected by the second optical flow sensor.

[0082] In some examples, the optical flow sensor can collect texture information (i.e., detection data) of the track every preset time interval and send the collected detection data to the control device. The optical flow sensor collects an image frame at a first time and sends it to the control device, and the control device determines the displacement of the carrying device based on the image frame and an image frame adjacent to the image frame (such as the previous image frame or the next image frame adjacent to the image frame).

[0083] For example, the odometer can be arranged on the motor of the walking wheel of the carrying device, for collecting pulse information (i.e., second detection information) of the motor to send to the control device, and the control device can determine the angle or distance of the walking wheel based on the pulse information of the walking wheel. During the movement of the carrying device, the odometer can collect second detection data, wherein the second detection data corresponds to the first detection data. For example, the second detection data can be the pulse information of the walking wheel collected by the odometer within the time range corresponding to the first detection data.

[0084] Step 520, determining a target image frame satisfying a preset condition in a plurality of initial image frames, and determining a first position of a first feature point in a first image frame and a second position of the first feature point in a second image frame adjacent to the first image frame.

[0085] In some examples, after the carrying device obtains the initial image frame collected by the image collection device, it can further determine whether the initial image frame satisfies the preset condition, so as to screen out invalid image frames that do not meet the requirements. That is, the preset condition can be used to screen valid image frames and discard invalid image frames.

[0086] For example, after receiving the image frame sent by the image acquisition device, the control device determines whether the image frame meets the preset condition. If the image frame meets the preset condition, the control device compares the image frame with a previous image frame, where the previous image frame is also an image frame meeting the preset condition. If the image frame does not meet the preset condition, the control device discards the image frame and reacquires a next image frame.

[0087] In some examples, the optical flow sensor pushes the actual displacement of the carrying device through the continuous displacement of the track surface texture, and thus the quality of the image frame collected by the optical flow sensor affects the reliability of subsequent displacement determination of the carrying device. The application filters effective image frames through the preset condition to avoid errors in subsequent displacement calculation of the carrying device caused by invalid image frames.

[0088] For example, the preset condition is related to the brightness of the image frame and / or the number of feature points on the image frame. After acquiring the initial image frames, the control device can filter the target image frames (i.e., effective image frames) through the brightness and / or the number of feature points of each initial image frame.

[0089] In some embodiments, the step 520 includes determining the brightness of each initial image frame and the number of feature points on each initial image frame; and determining, as the target image frame meeting the preset condition, an initial image frame in the plurality of initial image frames that has a brightness within a preset brightness value range and / or a number of feature points greater than a first preset number threshold.

[0090] In some examples, after acquiring the initial image frames, the brightness of the initial image frames and the number of feature points on the initial image frames can be determined, and whether the initial image frames meet the preset condition can be determined based on the brightness of the initial image frames and / or the number of feature points on the initial image frames.

[0091] For example, the initial image frame meeting the preset condition can include that the brightness of the initial image frame is within a preset brightness value range, or the number of feature points on the initial image frame is greater than a first preset number threshold, or the brightness of the initial image frame is within a preset brightness value range and the number of feature points on the initial image frame is greater than a first preset number threshold, which is not limited in the embodiments of the application. In order to further improve the subsequent slip detection accuracy, the embodiments of the application can determine the initial image frame with the brightness within the preset brightness value range and the number of feature points greater than the first preset number threshold as the target image frame.

[0092] In some examples, the brightness of the image frame can affect the identification and extraction of feature points in the image frame. When the brightness of the image frame is too low, the texture information of the track has insufficient contrast in the image frame, resulting in the subsequent inability to extract feature points corresponding to the texture information. When the brightness of the image frame is too high, the image frame will appear overblown, and the texture information of the track can be lost, also resulting in the subsequent inability to extract effectively identifiable feature points. Therefore, after obtaining the initial image frame, the brightness of the initial image frame can be determined, and the brightness can be compared with a preset brightness value range.

[0093] In some examples, the preset brightness value range can be determined by a brightness minimum value and a brightness maximum value. When the brightness of the image frame is greater than or equal to the brightness minimum value and less than or equal to the brightness maximum value, it indicates that the brightness of the image frame is within the preset brightness value range. The control device filters out the initial image frame with brightness within the preset brightness value range, thereby ensuring that the feature points in the image frame are clear and identifiable.

[0094] For example, the brightness minimum value and the brightness maximum value corresponding to the preset brightness value range can be set according to requirements, or can be determined according to historical data, which is not limited by the embodiments of the present application. For example, the brightness minimum value can be set to 50 Lux, and the brightness maximum value can be set to 200 Lux, that is, the initial image frame with brightness greater than or equal to 50 Lux and less than or equal to 200 Lux is the target image frame.

[0095] In some examples, the feature points on the image frame are key markers for subsequent determination of the displacement of the carrying device, and the feature points are used to represent the texture information on the driving path of the carrying device. Taking the driving of the carrying device on the track as an example, if the number of feature points on the surface of the track is small, there can be a problem of large error when calculating the displacement of the carrying device by using fewer feature points. Therefore, in order to improve the slip detection accuracy, a first preset number threshold can be set to limit the number of feature points in the image frame, thereby eliminating the initial image frame with insufficient number of feature points, and ensuring that sufficient markers are available for subsequent optical flow tracking.

[0096] For example, the first preset number threshold can be set according to actual requirements, or can be determined according to historical data, which is not limited by the embodiments of the present application. For example, the first preset number threshold can be set to 50 per frame, or can be set to 100 per frame. That is, when the number of feature points in the initial image frame is less than the first preset number threshold, the initial image frame is discarded, and the next image frame is reacquired.

[0097] In some examples, taking the example of an initial image frame satisfying preset conditions, including the brightness of the initial image frame being within a preset brightness value range and the number of feature points on the initial image frame being greater than a first preset number threshold, after acquiring the initial image frame, the control device, if determining that the brightness of the initial image frame is greater than or equal to a minimum brightness value and less than or equal to a maximum brightness value, further determines the number of feature points in the initial image frame. If the number of feature points in the initial image frame is greater than the first preset number threshold, the initial image frame is determined as a target image frame. After screening and obtaining the target image frame, the control device determines the displacement of the same feature point in two adjacent target image frames through optical flow tracking.

[0098] For example, after continuously determining two target image frames, the control device can further determine the displacement of the transport device based on the two target image frames. For example, the two target image frames include a first image frame and a second image frame. The first image frame and the second image frame can be two adjacent image frames, wherein the first image frame can be the next image frame of the second image frame or the previous image frame of the second image frame. This embodiment of the present application is not limited to this. This embodiment of the present application uses the example of the first image frame being the next image frame of the second image frame for illustrative purposes.

[0099] In some examples, the target image frame includes multiple initial feature points, which may include noise points or invalid feature points. To ensure the accuracy of subsequent calculations, the feature points in each target image frame need to be screened to obtain valid feature points (also referred to as target feature points) in each target image frame. The following describes the process of screening target feature points in the target image frame.

[0100] Figure 6 This is a schematic diagram of another method for detecting slippage of a handling device provided in an embodiment of the present application. Figure 6 As shown, before determining the first position of the first feature point in the first image frame in step 520 and the second position of the first feature point in the second image frame adjacent to the first image frame, the method further includes steps 610 to 620 as shown below.

[0101] Step 610: Determine a plurality of first initial feature points in the first image frame and a plurality of second initial feature points in the second image frame.

[0102] In some examples, the control device can determine the pixel displacement of the same feature point in two adjacent target image frames through an optical flow tracking algorithm (such as Farneback), thereby further determining the displacement of the handling equipment. In order to ensure the accuracy of subsequent displacement determination, the control device needs to remove noise points from the target image frame and retain valid feature points. Among them, the optical flow tracking algorithm can compare the pixel grayscale changes of the same feature point in two adjacent target image frames, thereby inferring the movement of the pixel of the feature point between the two image frames, such as the movement direction and distance.

[0103] In some examples, the control device can identify multiple initial feature points in each target image frame. Taking the target image frame including a first image frame and a second image frame as an example, the first image frame includes multiple first initial feature points, and the second image frame includes multiple second initial feature points.

[0104] In some embodiments, before step 610, the process includes: performing block processing on multiple initial feature points of the target image frame to divide the target image frame into multiple grid units; and performing screening processing on the grid units including the multiple initial feature points so that the number of initial feature points in the grid units is less than or equal to a second preset number threshold.

[0105] In some examples, some areas in the target image frame may have a relatively dense distribution of feature points, resulting in a clustering phenomenon. The dense feature points may cause computational redundancy in subsequent optical flow tracking. Therefore, when screening feature points in the target image frame, the present application needs to remove dense points in addition to removing noise points. The present application suppresses the clustering phenomenon of feature points by preprocessing the target image frame, ensures a uniform spatial distribution of feature points in the target image frame, and improves the computational efficiency of subsequent processing. For example, the preprocessing of the target image frame may include block processing and screening processing.

[0106] For example, after determining the target image frame, the target image frame can be segmented to divide the target image frame into a plurality of grid units. Each grid unit may or may not include feature points, and the number of feature points in each grid unit may be the same or different. When each grid unit includes a feature point, the number of feature points may be one or more.

[0107] For example, after determining the target image frame, the control device can divide the target image frame into grid units according to a preset size. The multiple grid units can cover the horizontal (width direction) and vertical (movement direction) surfaces of the track. The preset size can be set as needed, such as 5 cm × 5 cm. The specific division of the grid units is not limited in this embodiment of the application.

[0108] Exemplarily, for a grid cell including more feature points, the plurality of feature points in the grid cell can be subjected to a screening process, so as to avoid dense distribution of feature points in the same grid cell, thereby causing subsequent redundant calculation. For example, when the number of initial feature points in the grid cell exceeds an upper limit of the number of feature points, the plurality of initial feature points in the grid cell can be screened to screen the number of initial feature points in the grid cell to a second preset number threshold. The upper limit of the number of feature points and the second preset number threshold can be set based on demand or based on historical data, and the embodiments of the present application do not limit this.

[0109] In some examples, the screening process can be performed based on the feature intensity of each initial feature point in the grid cell. For example, the initial feature points with larger feature intensity can be retained, and the initial feature points with smaller feature intensity can be discarded, so as to ensure that the number of feature points in each grid cell is less than or equal to the second preset number threshold.

[0110] It should be noted that, in order to ensure uniform distribution of feature points in the target image frame, for the case that there is no feature point or the number of feature points in the grid cell is small, the control device can also process by interpolation or adjacent grid compensation, or it can also not need to be filled, and the embodiments of the present application do not limit this.

[0111] In some examples, after the above-mentioned preprocessing (i.e., the blocking processing and the screening processing) of the target image frame, the first initial feature points in the first image frame and the second initial feature points in the second image frame in the target image frame can be further determined. That is, the first initial feature points and the second initial feature points are the initial feature points after removing dense points.

[0112] In step 620, at least one target feature point is determined based on the first initial feature points and the second initial feature points.

[0113] In some examples, after determining the plurality of first initial feature points in the first image frame and the plurality of second initial feature points in the second image frame, at least one target feature point in the first image frame and the second image frame can be further screened. The at least one target feature point is located in the first image frame and the second image frame at the same time, and the at least one target feature point includes the first feature point.

[0114] That is, the control device can determine the feature points (i.e., target feature points) that exist in the first image frame and the second image frame at the same time, so as to remove noise points, and facilitate subsequent further determination of the displacement of the same feature point.

[0115] In some embodiments, step 620 comprises: determining, based on the first initial feature point and the second initial feature point, a plurality of first candidate feature points that are located in both the first image frame and the second image frame; determining a first candidate position of each first candidate feature point in the first image frame and a second candidate position of each first candidate feature point in the second image frame; determining a displacement difference between the first candidate position and the second candidate position of each first candidate feature point; and determining, based on the displacement difference, at least one target feature point from the plurality of first candidate feature points.

[0116] In some examples, the control device determines a plurality of first candidate feature points that are located in both the first image frame and the second image frame from the plurality of first initial feature points in the first image frame and the plurality of second initial feature points in the second image frame. After determining the first candidate feature points, the control device can further determine a position of each first candidate feature point in the first image frame and the second image frame, respectively.

[0117] For example, the position of a feature point in an image frame can be represented in the form of a two-dimensional coordinate position. For example, the first candidate position of a first candidate feature point in the first image frame can be (x1, y1), and the second candidate position of the first candidate feature point in the second image frame can be (x2, y2).

[0118] In some examples, after determining the first candidate position and the second candidate position, the control device further determines a displacement difference between the first candidate position and the second candidate position. For example, the displacement difference can be represented as (△x,△y), where△x = |x1-x2| and△y = |y1-y2|.

[0119] In some examples, after determining the displacement difference corresponding to each first candidate feature point, the control device determines a target feature point from the plurality of first candidate feature points based on the displacement difference.

[0120] In some embodiments, determining, based on the displacement difference, at least one target feature point from the plurality of first candidate feature points comprises: determining, from the plurality of first candidate feature points, a first candidate feature point whose displacement difference reaches a preset displacement difference threshold as the target feature point.

[0121] In some examples, when the track encounters abnormal situations such as light flickering or texture blurring (such as water stain covering), and the like, the displacement difference value of the feature points in the collected image frame will have a large error. For example, when the light flickering occurs, the overall brightness of the two image frames changes greatly, and the displacement difference value of some feature points can be randomly distributed; or when the texture blurring occurs, the edges of the feature points can appear to be flattened, and the subsequent algorithm can misjudge the displacement difference value. Such feature points can be referred to as feature points with unclear displacement changes (also referred to as abnormal feature points). In order to ensure the accuracy of subsequent displacement calculation, the abnormal feature points can be discarded, and the effective feature points (i.e., target feature points) can be retained.

[0122] For example, the preset displacement difference threshold value can be set according to requirements. For example, the preset displacement difference threshold value can be set to 0.5 pixels. When the displacement difference value between the first candidate position corresponding to the first candidate feature point and the second candidate position exceeds 0.5 pixels, it indicates that the first candidate feature point is a target feature point.

[0123] In some embodiments, the method further includes: determining the number of first candidate feature points and the number of second candidate feature points; and determining the proportion of abnormal feature points based on the number of first candidate feature points and the number of second candidate feature points. The second candidate feature points are the first candidate feature points in the plurality of first candidate feature points, and the displacement difference value of the first candidate feature points does not reach the preset displacement difference threshold value.

[0124] In some examples, after the target feature points are determined, the candidate feature points in the plurality of first candidate feature points except the target feature points are referred to as second candidate feature points, and the second candidate feature points can also be referred to as abnormal feature points. The control device determines the number of first candidate feature points and the number of second candidate feature points.

[0125] For example, the proportion of abnormal feature points can be determined based on the number of first candidate feature points and the number of second candidate feature points. The number of second candidate feature points is the number of first candidate feature points minus the number of target feature points, and the proportion of abnormal feature points can be the ratio of the number of second candidate feature points to the number of first candidate feature points.

[0126] In some examples, the higher the proportion of abnormal feature points, the higher the number of abnormal feature points in the target image frame, and the higher the number of abnormal feature points indicates that the detection data corresponding to the target image frame is unreliable and has more noise. Therefore, the target image frame needs to be discarded to avoid interfering with the subsequent judgment. When the proportion of abnormal feature points is lower, it indicates that the number of abnormal feature points in the target image frame is smaller, and the target image frame is an effective image frame.

[0127] In some examples, after determining the proportion of abnormal feature points, the proportion of abnormal feature points can be compared with a preset proportion threshold. In the case that the proportion of abnormal feature points is less than the preset proportion threshold, the following step 530 is continued to execute; in the case that the proportion of abnormal feature points is greater than or equal to the preset proportion threshold, it indicates that the current target image frame does not meet the requirements and will affect the subsequent calculation, and the target image frame can be discarded from the image frame collected by the image acquisition device.

[0128] For example, in the case that the proportion of abnormal feature points in the first image frame and its adjacent second image frame is greater than or equal to the preset proportion threshold, the first image frame can be discarded, and the next image frame and the image frame adjacent to the next image frame are re-collected, and the feature points in the two adjacent image frames are judged as described above, and the judgment manner is similar to the above process (such as step 520). To avoid repetition, details are not repeated here. It should be noted that the target image frame in the embodiment of the application is an image frame that meets the above requirements.

[0129] Step 530, based on the first position and the second position, determining a first detection distance corresponding to the image acquisition device, and based on the second detection data, determining a second detection distance corresponding to the odometer.

[0130] For example, after determining the at least one target feature point, the positions of each target feature point in the first image frame and the second image frame are determined. For example, the first position of a first feature point in the first image frame and the second position of the first feature point in the second image frame are determined in the at least one target feature point, so as to determine the first detection distance corresponding to the image acquisition device. Wherein, the first feature point is any feature point in the at least one target feature point. That is, the first detection distance is the first detection distance of the conveying device calculated by the image acquisition device (i.e. the first optical flow sensor or the second optical flow sensor) through the pixel displacement of the track texture, that is, the actual displacement of the conveying device.

[0131] In some examples, after determining the first detection distance, the second detection data of the walking wheel (such as the first walking wheel or the second walking wheel) acquired by the odometer during the first image frame and the second image frame can be determined. After acquiring the second detection data, the control device determines the second detection distance corresponding to the conveying device based on the second detection data, that is, the theoretical displacement of the conveying device. For example, the first detection data collected by the optical flow sensor also includes the time stamp corresponding to each image frame, and the control device can determine the second detection data corresponding to the odometer based on the time stamp corresponding to the first image frame and the second image frame.

[0132] Step 540, based on the first detection distance and the second detection distance, determining the slipping condition of the conveying device.

[0133] In some examples, due to the cumulative error of the odometer, there is usually a certain difference between the actual displacement corresponding to the optical flow sensor and the theoretical displacement corresponding to the odometer. The distance difference between the first detection distance and the second detection distance can be used to determine whether the carrying device slips.

[0134] In some embodiments, step 540 includes: determining a first distance difference between the first detection distance and the second detection distance when the proportion of abnormal feature points is less than the preset proportion threshold; determining that the carrying device slips if the first distance difference exceeds a target distance threshold; or determining that the carrying device slips if the first distance difference exceeds the target distance threshold and a duration for which the first distance difference exceeds the target distance threshold exceeds a target time threshold.

[0135] In some examples, when the proportion of abnormal feature points is less than the preset proportion threshold, it indicates that the first image frame and the second image frame are valid image frames, and the first distance difference between the first detection distance and the second detection distance can be further determined.

[0136] For example, if the proportion of abnormal feature points is small, such as less than the preset proportion threshold, it indicates that there is less noise in the first image frame and the second image frame, the detection data is reliable, the first detection distance calculated is accurate, and thus the first distance difference obtained also has reference value. If the proportion of abnormal feature points is small, such as greater than or equal to the preset proportion threshold, it indicates that there is more noise, the first detection distance calculated may be caused by noise rather than real slip, resulting in a large error in judgment. Therefore, in order to ensure the accuracy of slip detection, only when the proportion of abnormal feature points is less than the preset proportion threshold, the first distance difference between the first detection distance and the second detection distance is used to determine whether the carrying device slips, thereby improving the accuracy of slip detection.

[0137] For example, if the first distance difference exceeds the target distance threshold, or the first distance difference exceeds the target distance threshold and the duration for which the first distance difference exceeds the target distance threshold exceeds the target time threshold, it can be determined that the carrying device slips.

[0138] In some examples, when the carrying device does not slip, there is no relative sliding between the walking wheels of the carrying device and the track, the actual displacement corresponding to the optical flow sensor (i.e., the first detection distance) should be equal to the theoretical displacement corresponding to the odometer (i.e., the second detection distance), or the deviation between the actual displacement corresponding to the optical flow sensor and the theoretical displacement corresponding to the odometer is within an error range. When the carrying device slips, the walking wheels of the carrying device will idle, the odometer will still determine the theoretical displacement according to the number of rotations of the walking wheels, resulting in a large deviation between the theoretical displacement and the actual displacement corresponding to the optical flow sensor, and the theoretical displacement will be significantly greater than the actual displacement. For example, when the carrying device slips, the walking wheels of the carrying device rotate 10 times, the theoretical displacement is 1 meter, and the actual displacement corresponding to the optical flow sensor is only 0.8 meters. In this case, it can be determined that the carrying device has slipped.

[0139] Exemplarily, after determining the first distance difference value, the first distance difference value can be compared with a target distance threshold value, and when the first distance difference value exceeds the target distance threshold value, it can be determined that the carrying device has slipped. In order to further ensure the accuracy of the judgment, the control device can continue to determine the distance difference value corresponding to the next two adjacent image frames according to the above steps 510 to 540, and if the distance difference value also exceeds the target distance threshold value, the distance difference value corresponding to the next two image frames is continued to be determined, and so on, until the duration reaches the target time threshold value. That is, during the duration reaches the target time threshold value, if the distance difference value corresponding to each group of two image frames exceeds the target distance threshold value, it can be determined that the carrying device has slipped.

[0140] In some examples, the duration can start timing from the first time the distance difference value (such as the first distance difference value) exceeds the target distance threshold value, and stop timing when the duration reaches the target time threshold value, and start timing again when the distance difference value exceeds the target distance threshold value next time. That is, the control device can start timing when it is determined that the first distance threshold exceeds the target distance threshold value, and determine that the distance difference value corresponding to the subsequent multiple groups of image frames exceeds the target distance threshold value during the duration reaches the target time threshold value, and then determine that the carrying device has slipped.

[0141] For example, if the distance difference between the distance detected by the optical flow sensor and the distance detected by the odometer is d1, and d1 exceeds the target distance threshold, the control device starts timing and continues to determine the distance difference d2 between the next image frame c corresponding to image frame b and image frame b. If d2 also exceeds the target distance threshold and the duration has not reached the target time threshold, the control device continues to determine the distance difference d3 between the next image frame d corresponding to image frame c and image frame c. If d2 also exceeds the target distance threshold and the duration reaches the target time threshold, it is determined that the carrying device slips. It should be noted that if there is a group of image frames (if image frame d and image frame c) corresponding to the distance difference less than the target distance threshold before the duration reaches the target time threshold, it is determined that the carrying device does not slip and the timing is stopped.

[0142] In some examples, the judgment condition for determining that the carrying device slips includes a target distance threshold. If the first distance difference does not exceed the target distance threshold, it is determined that the carrying device does not slip. In another example, the judgment condition for determining that the carrying device slips includes a target distance threshold and a target time threshold. If the first distance difference exceeds the target distance threshold and the duration for which the first distance difference exceeds the target distance threshold does not exceed the target time threshold, it is also determined that the carrying device does not slip.

[0143] The carrying device slip detection method provided by the embodiments of the present application can avoid the situation that the distance difference temporarily exceeds the target distance threshold due to dynamic transient interference such as slight track bumps or sensor transient noise, eliminate occasional interference, ensure that the slip is a persistent abnormal state, and further improve the accuracy of slip judgment.

[0144] In some examples, the target distance threshold and the target time threshold can be set to fixed values according to requirements. For example, the target distance threshold can be set to 0.5 cm, and the target time threshold can be set to 200 ms. Alternatively, the target distance threshold and the target time threshold can also be dynamically adjusted according to the specific working conditions of the warehouse, which is not limited in the embodiments of the present application. The following describes the case where the target distance threshold and the target time threshold can be dynamically adjusted.

[0145] In some embodiments, the method further includes determining the friction coefficient between the walking wheel of the carrying device and the track, and the carrying state of the carrying device; and determining the target distance threshold and the target time threshold based on the friction coefficient and / or the carrying state.

[0146] Exemplarily, the distance threshold and the time threshold can be related to the carrying state of the carrying device and / or the friction coefficient. According to the track friction coefficient and / or the load change, the control device can adaptively adjust the distance threshold and the time threshold to obtain a target distance threshold and a target time threshold, so as to avoid misjudgment caused by fixed thresholds.

[0147] For example, a plurality of distance thresholds and a plurality of time thresholds can be stored in the control device, and the control device can select a target distance threshold most matched from the plurality of distance thresholds and a target time threshold most matched from the plurality of time thresholds according to the track friction coefficient and / or the load change.

[0148] In some examples, the control device can determine the target distance threshold and the target time threshold based on the friction coefficient, or determine the target distance threshold and the target time threshold based on the carrying state, or determine the target distance threshold and the target time threshold based on the friction coefficient and the carrying state. The embodiments of the present application are not limited in this regard.

[0149] It should be noted that the control device can only determine the target distance threshold based on the friction coefficient and / or the carrying state, and the target time threshold is fixedly set according to requirements, or the control device can determine the target distance threshold and the target time threshold based on the friction coefficient and / or the carrying state. The embodiments of the present application exemplarily illustrate the determination of the target distance threshold and the target time threshold.

[0150] In some examples, the friction coefficient between the walking wheel of the carrying device and the track is related to the material of the track, the wear degree of the track, the storage environment, etc. For example, the friction coefficients of metal tracks and wooden tracks are different, the friction coefficients can be different when the wear degrees of the tracks are different, and the friction systems corresponding to dry and wet storage environments are also different. When the friction coefficient changes, the distance threshold and the time threshold can be adjusted correspondingly based on the friction coefficient.

[0151] For example, the friction coefficient can be in a positive proportional relationship with the distance threshold and the time threshold, that is, the smaller the friction system is, the smaller the distance threshold is, and the shorter the time threshold is; the larger the friction system is, the larger the distance threshold is, and the longer the time threshold is.

[0152] In some examples, the carrying device moves on the track by the walking wheel to perform a carrying task, and the carrying state of the carrying device can be an empty load state or a load state during the movement of the carrying device on the track. The distance threshold and the time threshold corresponding to the carrying state of the carrying device can be different.

[0153] For example, the distance threshold and the time threshold are longer when the carrying device is in an empty state, and the distance threshold and the time threshold are shorter when the carrying device is in a loaded state. When the carrying device is in a loaded state, the driving force of the traveling wheel of the carrying device is greater, and the risk of slipping is higher, so the distance threshold and the time threshold need to be shortened. When the carrying state is an empty state, the carrying device is less likely to slip, so the distance threshold and the time threshold can be appropriately increased.

[0154] It should be noted that the distance threshold and the time threshold can also be related to the moving speed and acceleration of the carrying device. For example, when the traveling speed of the carrying device is high, the same degree of slipping will result in a greater distance difference, so the distance threshold and the time threshold can be increased. The present application does not make specific limitations in this regard.

[0155] For example, the control device can create a threshold correspondence relationship of the distance threshold and the time threshold corresponding to different working condition scenarios. The threshold correspondence relationship can include the distance threshold and the time threshold corresponding to different working condition scenarios, respectively. After determining the working condition scenario corresponding to the carrying device, the control device queries the threshold correspondence relationship to determine the target distance threshold and the target time threshold corresponding to the carrying device.

[0156] In some examples, the threshold correspondence relationship can include the distance threshold and the time threshold corresponding to different friction coefficient ranges and / or different carrying states.

[0157] For example, in the threshold correspondence relationship, the friction coefficient range A1 corresponds to the distance threshold D1 and the time threshold T1, the friction coefficient range A2 corresponds to the distance threshold D2 and the time threshold T2; the empty state corresponds to the distance threshold D3 and the time threshold T3, and the loaded state corresponds to the distance threshold D4 and the time threshold T4; the friction coefficient range A1 and the empty state correspond to the distance threshold D4 and the time threshold T4, the friction coefficient range A1 and the loaded state correspond to the distance threshold D5 and the time threshold T5; the friction coefficient range A2 and the empty state correspond to the distance threshold D6 and the time threshold T6, and the friction coefficient range A2 and the loaded state correspond to the distance threshold D7 and the time threshold T7. When the control device determines that the friction coefficient between the carrying device and the track is in the friction coefficient range A1 and the carrying device is in an empty state, the target distance threshold can be determined as the distance threshold D4, and the target time threshold can be determined as the time threshold T4.

[0158] The carrying device slip detection method provided by the present application can dynamically and adaptively adjust the distance threshold and the time threshold based on the carrying state and / or the friction coefficient between the carrying device and the track, thereby avoiding false judgments caused by fixed thresholds.

[0159] In some embodiments, in a case where it is determined that the carrying device slips, the control output device outputs alarm information, and adjusts the position of the carrying device based on the first detection distance.

[0160] In some examples, when it is determined that the carrying device slips, the control device can control the output device in the warehouse system to output alarm information, and calibrate the pose of the carrying device to avoid accidents or positioning failure.

[0161] For example, the output device can be a display device, an audio device or other alarm device, and the alarm information can include information of the carrying device that slips and the position where the carrying device slips, and the like, which are not limited in the embodiments of the present application. The staff can determine the slipping position and the carrying device that slips according to the alarm information output by the output device.

[0162] In some examples, the control device can compensate the position of the carrying device based on the first detection distance corresponding to the optical flow sensor, so as to adjust the carrying device to the accurate position. For example, since the slipping of the carrying device can cause the detection data of the odometer to be higher, the control device can correct the current position of the carrying device according to the actual displacement of the optical flow sensor. For example, when the odometer determines that the moving distance of the carrying device is 1 meter, and the optical flow sensor determines that the moving distance of the carrying device is 0.8 meter, the carrying device needs to be positioned by subtracting an error of 0.2 meter from the positioning result.

[0163] The carrying device slipping detection method provided by the embodiments of the present application detects the slipping of the carrying device (such as a four-way vehicle) through an image acquisition device (such as an optical flow sensor) and an odometer. The optical flow sensor can acquire image frames corresponding to the identification information (such as the texture of the track) on the track during the movement of the four-way vehicle. The control device determines the positions of the same feature points in adjacent image frames, so as to determine the first detection distance (such as the actual displacement) of the carrying device. Then, the second detection distance (such as the theoretical displacement) of the carrying device is determined through the detection data of the odometer. Finally, the actual displacement corresponding to the optical flow sensor and the theoretical displacement corresponding to the odometer are combined to determine whether the carrying device slips. The embodiments of the present application realize the detection of whether the carrying device slips through the cooperation of the optical flow sensor and the odometer. Since the detection of the optical flow sensor depends on the texture information of the track itself, the track does not need to be modified and is not affected by the environment, and the practicality and detection accuracy are high. Therefore, the embodiments of the present application can realize high-precision, low-cost and high-reliability slipping detection of the four-way vehicle in a complex environment.

[0164] In some examples, the track in the warehouse system (such as the first track or the second track) can be connected by multiple sub-tracks. For some sub-tracks, the texture thereon is clear and identifiable, so that the image frames collected are valid image frames, and whether the carrying device slips can be determined through the above steps 520 to 540. For some sub-tracks, the surface texture thereof can be less or cannot be effectively identified, so that the number of feature points on the image frames continuously collected by the optical flow sensor cannot meet the requirement of subsequent displacement calculation. For example, when the number of feature points on the image frames corresponding to the sub-tracks are all less than the first preset number threshold, the control device determines that the image frames are all invalid image frames, so that the slip of the carrying device on the sub-tracks cannot be determined.

[0165] To solve this problem, in addition to determining the slip of the carrying device based on the feature points, the slip of the carrying device can also be determined based on the connection point (also referred to as the reference point) between two sub-tracks according to the embodiments of the present application. The process of determining the slip of the carrying device based on the connection point between two sub-tracks will be described below. Figure 7 The process of determining the slip of the carrying device based on the connection point between two sub-tracks will be described below.

[0166] Figure 7 Another schematic diagram of a method for detecting the slip of a carrying device is provided according to the embodiments of the present application. As shown in the figure, after the step 510, the method further includes steps 710 to 740. Figure 7

[0167] Step 710: Determine the number of feature points in a third image frame in the plurality of initial image frames.

[0168] In some examples, the plurality of initial image frames can include the third image frame, and the third image frame is any image frame in the plurality of initial image frames. After the control device acquires the third image frame, the number of feature points in the third image frame is determined.

[0169] Step 720: In the case that the number of feature points in the third image frame is less than a third preset number threshold, determine whether the first reference point is included in the third image frame.

[0170] In some examples, the third preset number threshold can be set according to requirements. For example, the third preset number threshold can be less than or equal to the first preset number threshold. For example, the third preset number threshold can be set to 50 / frames, or 20 / frames, which is not limited in the embodiments of the present application. When the number of feature points in the third image frame is less than the third preset number threshold, it indicates that the number of feature points in the third image frame is small, which does not meet the requirement of subsequent calculation, so that whether the first reference point is included in the third image frame can be further determined.

[0171] In some embodiments, the reference point is used to represent the connection mark between two sub-tracks.​

[0172] Figure 8 A schematic diagram of a carrying device provided by an embodiment of the present application moving on a track.

[0173] As shown in Figure 8 , the carrying device 20 travels on the track a, the track a is connected by the sub-track a1, the sub-track a2 and the sub-track a3, the connection point between the sub-track a1 and the sub-track a2 is the reference point b1, and the connection point between the sub-track a2 and the sub-track a3 is the reference point b2.

[0174] As shown in Figure 8 , when the sub-track a1, the sub-track a2 and the sub-track a3 are low-texture tracks, the number of feature points in the image frames collected by the carrying device 20 during the movement of the sub-track a1 to the sub-track a3 may be less than the third preset number threshold. In this case, the control device can determine whether the reference point is included in each image frame. For example, when the carrying device 20 moves to the first position, the image frame (such as the third image frame) collected by the optical flow sensor includes the reference point b1. The control device records the reference point b, and the time stamp corresponding to the third image frame.

[0175] Step 730, in the case that the third image frame includes the first reference point, if the image acquisition device collects a fourth image frame including a second reference point, the control device determines the third detection distance corresponding to the image acquisition device based on the first reference point and the second reference point, and determines the fourth detection distance corresponding to the odometer.

[0176] In some examples, in the case that the number of feature points in the third image frame is less than the third preset number threshold, and the first reference point is included in the feature points, the control device records the first time stamp at which the third image frame is collected, and continues to acquire subsequent multiple image frames collected by the optical flow sensor. If the number of feature points in the multiple image frames subsequently collected by the optical flow sensor continues to be less than the third preset number threshold, the control device determines whether the second reference point is included in each image frame. Until the fourth image frame including the second reference point is collected by the optical flow sensor, the control device records the second time stamp at which the fourth image frame is collected.

[0177] Exemplarily, the control device can determine the sub-track corresponding to the first reference point and the sub-track corresponding to the second reference point based on the position of the first reference point and the position of the second reference point, so as to determine the third detection distance between the third image frame and the fourth image frame.

[0178] In some embodiments, the third detection distance is the length of the sub-track between the first reference point and the second reference point.

[0179] As shown in Figure 8As shown, taking the first reference point as reference point b1 and the second reference point as reference point b2 as an example, since the first reference point is located at the connection position between the sub-track a1 and the sub-track a2, and the second reference point is located at the connection position between the sub-track a2 and the sub-track a3, the third detection distance is the length of the sub-track a2.

[0180] In some examples, the lengths of the sub-tracks can be pre-stored in the control device, and after the positions of the reference points are determined, the length of the sub-track between two adjacent reference points can be determined, so as to further determine the third detection distance corresponding to the optical flow sensor.

[0181] In some examples, after the third detection distance (i.e., the actual displacement of the carrying device) is determined, the detection data of the odometer between the third image frame and the fourth image frame can be determined based on the first timestamp and the second timestamp, and the fourth detection distance (i.e., the theoretical displacement of the carrying device) corresponding to the detection data can be determined.

[0182] In step 740, a second distance difference between the third detection distance and the fourth detection distance is determined, and it is determined that the carrying device slips when the second distance difference is less than a target distance threshold.

[0183] For example, after the third detection distance and the fourth detection distance are determined, a second distance difference between the third detection distance and the fourth detection distance is determined, and the second distance difference is compared with a target distance threshold. In the case where the second distance difference exceeds the target distance threshold, it is determined that the carrying device slips; in the case where the second distance difference does not exceed the target distance threshold, it is determined that the carrying device does not slip.

[0184] It should be noted that since the sub-track has a certain length, the time length of the optical flow sensor between the first reference point and the second reference point can have exceeded the target time threshold, and therefore in this case, only the second distance difference needs to be compared with the target distance threshold, without determining the relationship between the continuous time length and the target time threshold.

[0185] The carrying device slip detection method provided by the embodiments of the present application can determine the actual displacement of the optical flow sensor based on the inherent and stable connection points (i.e., reference points) between the sub-tracks when there are few texture information on the track and there are not enough feature points for subsequent displacement calculation, so that the carrying device slip can also be further determined based on the actual displacement of the optical flow sensor and the theoretical displacement of the odometer. Therefore, the embodiments of the present application can be applied to carrying device slip detection in different texture scenes, and improve the applicability and reliability of the slip detection.

[0186] Figure 9 A schematic diagram of a carrying device slip detection process provided by the embodiments of the present application is shown. It should be noted that,Figure 9 Take the four-way vehicle as the carrying device and the optical flow sensor as the image acquisition device as an example. The following describes the processing of the image frames collected by the optical flow sensor. Figure 9 The processing of the image frames collected by the optical flow sensor is described.

[0187] As shown in FIG. 1, the four-way vehicle uses a high-resolution optical flow sensor to collect image frames corresponding to the texture information of the track surface and sends them to the controller. Figure 9

[0188] The controller performs multi-size feature detection on the image frames to ensure the robustness of the feature points in the collected image frames to changes in light and viewing angle. For example, the controller can use a multi-scale Smallest Univalue Segment Assimilating Nucleus (SUSAN) operator or an improved Harris corner detector to extract stable feature points on the track surface at different resolutions. The SUSAN operator is suitable for low-contrast texture areas and detects features such as circular points and edges. The Harris corner detector is suitable for geometric features such as track joints and right-angle markers and locates corner points through a response function. After extracting the feature points in the image frames, the feature points can be analyzed and filtered to eliminate invalid points, redundant points, and noise points with blurred edges, and to retain valid feature points.

[0189] After completing the filtering of the feature points, the controller can perform block processing on the image frames. The controller can divide the image frames corresponding to the track surface into regular grids to avoid local feature clustering and ensure uniform spatial distribution of the feature points, thereby improving the computational efficiency of subsequent processing. For example, the image frames can be divided into grid cells with a fixed size (e.g., 5 cm x 5 cm), and for each grid cell, the feature strength (e.g., the response value of the SUSAN operator or the response function value of the Harris corner) of all feature points is calculated, and the feature points with higher feature strength are retained. If there are no feature points or the feature strength is low in the grid, it is marked and compensated by interpolation or neighboring grids in the subsequent processing. Through block processing, the dense distribution of feature points in the same area (clustering phenomenon) can be avoided, the computational redundancy of subsequent optical flow tracking can be reduced, and the representativeness of the track surface features can be ensured.

[0190] After completing the block processing, the controller can perform sub-pixel positioning processing, i.e., through weighted averaging of multi-scale feature point coordinates, to achieve sub-pixel level precision positioning and ensure an error less than a preset pixel threshold (e.g., 0.1 pixels) to meet high-precision requirements. For example, the weighted mean value can be calculated for the pixel coordinates (x1, y1), (x2, y2), …, (xn, yn) of the same feature point in consecutive multiple frames (e.g., n frames) of images.

[0191] ​The controller can pre-construct a feature library and store the feature library. The feature library can store each marker point on the track and the position of the marker point on the track. After obtaining the image frame, the controller can query whether the feature points in the feature library exist in the image frame and the positions of the feature points in the feature library. The controller can also dynamically update the feature library to support real-time input of new feature points in the image frame, facilitating faster matching in the future.

[0192] After determining each feature point in the image frame and the position of each feature point, the controller can perform optical flow tracking and use an optical flow algorithm to track the displacement of the feature points. Subsequently, in combination with the actual displacement detected by the odometer, it is determined whether the four-way vehicle has slipped.

[0193] Figure 10 Another schematic diagram of a slip detection process of a carrying device is provided for the embodiments of the present application. It should be noted that, Figure 10 Taking the carrying device as a four-way vehicle and the image acquisition device as an optical flow sensor as an example. The process of determining that the carrying device has slipped will be described below. Figure 10

[0194] Step 1010, obtaining an image frame collected by an optical flow sensor.

[0195] For example, the control device (such as a controller) of the carrying device can obtain the image frame of the texture information on the track collected by the optical flow sensor in real time. It should be noted that step 1010 is similar to “obtaining first detection data collected by the image acquisition device” in step 510 described above, and details are not repeated here to avoid repetition.

[0196] Step 1020, image frame screening.

[0197] For example, the control device can screen the image frames based on the brightness of each image frame, discard image frames with insufficient brightness or excessively high brightness, and screen target image frames.

[0198] Step 1030, feature point detection and screening.

[0199] For example, the control device can detect and extract feature points on the screened image frames (i.e., target image frames). After extracting the feature points, the feature points are screened to remove feature points with low contrast and dense feature points.

[0200] Step 1040, optical flow tracking to determine the actual displacement.

[0201] For example, the control device can determine the displacement of the same feature point in two consecutive image frames based on the screened feature points, and thus calculate the actual displacement S1.

[0202] ​Step 1050, reading the odometer data to determine the theoretical displacement.

[0203] For example, the controller reads the detection data of the odometer, and calculates the theoretical displacement S2 according to the detection data of the odometer.

[0204] Step 1060, calculating the displacement difference.

[0205] For example, the control device can calculate the displacement difference ΔS according to the actual displacement S1 corresponding to the optical flow sensor and the theoretical displacement S2 corresponding to the odometer, ΔS = |S1-S2|.

[0206] Step 1070, determining whether the carrying device slips.

[0207] For example, after obtaining the displacement difference, the control device can determine whether the carrying device slips based on a dynamic threshold. The dynamic threshold includes a distance threshold and a time threshold. When the displacement difference exceeds the distance threshold and the duration exceeds the time threshold, it is determined that the four-way vehicle slips, and an alarm is triggered.

[0208] Step 1080, path correction.

[0209] For example, after detecting the slip, the control device can correct the position deviation through fusion of the laser radar and the IMU data to ensure the accuracy of the navigation path.

[0210] Figure 11 A schematic diagram of a carrying device slip detection device provided by an embodiment of the present application is shown in FIG. 11. Figure 11 As shown in FIG. 11, the carrying device slip detection device 1100 includes an acquisition module 1110, a first determination module 1120, a second determination module 1130, and a third determination module 1140. Wherein:

[0211] The acquisition module 1110 is configured to: during movement of the carrying device along the driving path, acquire first detection data collected by an image collection device of the carrying device, and second detection data collected by an odometer of the carrying device; wherein the first detection data includes a plurality of initial image frames.

[0212] The first determination module 1120 is configured to: determine a target image frame that satisfies a preset condition in the plurality of initial image frames, and determine a first position of a first feature point in a first image frame and a second position of the first feature point in a second image frame adjacent to the first image frame; wherein the target image frame includes the first image frame and the second image frame.

[0213] The second determination module 1130 is configured to: determine a first detection distance corresponding to the image collection device based on the first position and the second position, and determine a second detection distance corresponding to the odometer based on the second detection data.

[0214] The third determination module 1140 is configured to determine the slipping condition of the carrying device based on the first detection distance and the second detection distance.

[0215] In some embodiments, the first determination module 1120 is configured to determine the brightness of each initial image frame and the number of feature points on each initial image frame; wherein the travel path of the carrying device includes a track, and the feature points are used to represent the identification information of the track; and the initial image frames in which the brightness is within a preset brightness value range and / or the number of feature points is greater than a first preset number threshold are determined as the target image frames that meet the preset condition.

[0216] In some embodiments, the first determination module 1120 is further configured to determine a plurality of first initial feature points in the first image frame and a plurality of second initial feature points in the second image frame; determine at least one target feature point based on the first initial feature points and the second initial feature points; wherein the at least one target feature point is located in both the first image frame and the second image frame, and the at least one target feature point includes the first feature point.

[0217] In some embodiments, the first determination module 1120 is configured to determine a plurality of first candidate feature points that are located in both the first image frame and the second image frame based on the first initial feature points and the second initial feature points; determine a first candidate position of each first candidate feature point in the first image frame and a second candidate position of each first candidate feature point in the second image frame; determine a displacement difference value between the first candidate position and the second candidate position corresponding to each first candidate feature point; and determine at least one target feature point from the plurality of first candidate feature points based on the displacement difference values.

[0218] In some embodiments, the first determination module 1120 is configured to determine, from the plurality of first candidate feature points, a first candidate feature point whose displacement difference value reaches a preset displacement difference threshold as the target feature point.

[0219] In some embodiments, the first determination module 1120 is further configured to determine the number of first candidate feature points and the number of second candidate feature points; wherein the second candidate feature points are first candidate feature points in the plurality of first candidate feature points whose displacement difference values do not reach the preset displacement difference threshold; and determine the proportion of abnormal feature points based on the number of first candidate feature points and the number of second candidate feature points.

[0220] In some embodiments, the third determining module 1140 is configured to: in a case where the proportion of the abnormal feature points is less than a preset proportion threshold, determine a first distance difference value between the first detection distance and the second detection distance; if the first distance difference value exceeds a target distance threshold, determine that the carrying device slips; or if the first distance difference value exceeds the target distance threshold and a duration for which the first distance difference value exceeds the target distance threshold exceeds a target time threshold, determine that the carrying device slips.

[0221] In some embodiments, the third determining module 1140 is further configured to: determine a friction coefficient between a walking wheel of the carrying device and the track, and a carrying state of the carrying device; wherein the carrying state includes an empty load state and a loaded state; and determine the target distance threshold and the target time threshold based on the friction coefficient and / or the carrying state.

[0222] In some embodiments, the first determining module 1120 is further configured to: in a case where the proportion of the abnormal feature points is greater than or equal to the preset proportion threshold, discard the target image frame and re-acquire the detection information of the image acquisition device.

[0223] In some embodiments, the first determining module 1120 is further configured to: perform block processing on the plurality of initial feature points of the target image frame to divide the target image frame into a plurality of grid units; and perform screening processing on the grid units including the plurality of initial feature points, so that the number of initial feature points in the grid units is less than or equal to a second preset number threshold.

[0224] In some embodiments, the device 1100 for determining that the carrying device slips further includes a fourth determining module configured to: determine the number of feature points in a third image frame of the plurality of initial image frames; in a case where the number of feature points in the third image frame is less than a third preset number threshold, determine whether the first reference point is included in the third image frame; in a case where the first reference point is included in the third image frame, if a fourth image frame including the second reference point is acquired by the image acquisition device, determine a third detection distance corresponding to the image acquisition device based on the first reference point and the second reference point, and determine a fourth detection distance corresponding to the odometer; determine a second distance difference value between the third detection distance and the fourth detection distance, and in a case where the second distance difference value exceeds the target distance threshold, determine that the carrying device slips.

[0225] In some embodiments, the carrying device moves on the track through the walking wheel, the track includes a plurality of sub-tracks, and the reference points are used to represent the connection identifier between two sub-tracks; and the detection distance corresponding to two adjacent reference points is the length of the sub-track between the two adjacent reference points.

[0226] In some embodiments, the carrying device includes first walking wheels and second walking wheels, the image acquisition device includes a first optical flow sensor arranged between the two first walking wheels, and a second optical flow sensor arranged between the two second walking wheels. The acquisition module 1110 is configured to: acquire detection information corresponding to the first track acquired by the first optical flow sensor during movement of the carrying device on the first track by the first walking wheels; or acquire detection information corresponding to the second track acquired by the second optical flow sensor during movement of the carrying device on the second track by the second walking wheels.

[0227] In some embodiments, the device 1100 for detecting skidding of the carrying device further includes a control module configured to: in a case where it is determined that the carrying device skids, control the output device to output alarm information, and adjust the position of the carrying device based on the first detection distance.

[0228] Figure 12 A schematic diagram of an electronic device is provided for the embodiments of the present application. In some embodiments, the electronic device includes one or more processors and a memory. The memory is configured to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the carrying device skidding detection method in the above embodiments.

[0229] As shown in Figure 12 The electronic device 1000 includes a processor 1001 and a memory 1002. The electronic device 1000 may

[0230] The processor 1001, the memory 1002, and the communication interface 1003 communicate with each other through the communication bus 1004. The communication interface 1003 is used to communicate with network elements such as clients or other servers.

[0231] In some embodiments, the processor 1001 is configured to execute the program 1005, and specifically can execute the related steps in the carrying device skidding detection method embodiments described above. Specifically, the program 1005 can include program code including computer executable instructions.

[0232] The processor 1001 may, for example, be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement one or more embodiments of the present application. The electronic device 1000 can include one or more processors of the same type, such as one or more CPUs, or one or more processors of different types, such as one or more CPUs and one or more ASICs.

[0233] In some embodiments, the memory 1002 is configured to store a program 1005. The memory 1002 can include a high-speed RAM memory, and can also include a non-volatile memory (NVM), such as at least one disk memory.

[0234] The program 1005 can specifically be invoked by the processor 1001 to cause the electronic device 1000 to perform the operations of the handling equipment slip detection method.

[0235] The computer-readable storage medium stores at least one executable instruction, which, when executed on the electronic device 1000, causes the electronic device 1000 to perform the handling equipment slip detection method in the above embodiments.

[0236] The executable instruction can specifically be used to cause the electronic device 1000 to perform the operations of the handling equipment slip detection method.

[0237] For example, the computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0238] In some embodiments, the computer program product includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions that, when executed by a computer, cause the computer to perform the handling equipment slip detection method described in any of the above embodiments.

[0239] In some embodiments, the computer program, when executed by a processor, can implement the scheduling method described in any of the above embodiments.

[0240] The beneficial effects that can be achieved by the moving device slip detection device, the electronic device, the computer readable storage medium, the computer program product and the computer program provided by the embodiments of the present application can refer to the beneficial effects of the moving device slip detection method provided in the above, and will not be described here again.

[0241] It should be noted that, in the application, the relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitation, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0242] Each of the embodiments in the specification is described in a relevant manner, and the same or similar parts between each of the embodiments can be referred to each other, and each of the embodiments focuses on the difference from other embodiments. Especially, for the device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can refer to the part of the method embodiments.

[0243] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered a list of executable instructions for implementing logic functions, and can be specifically embodied in any computer-readable medium for use by an instruction execution system, apparatus or device, such as a computer-based system, a system including a processor or other system that can fetch the instructions from the instruction execution system, apparatus or device and execute the instructions, or in conjunction with these instructions execution system, apparatus or device.

[0244] For the purpose of the present specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transport programs for use by or in connection with an instruction execution system, apparatus or device, or in conjunction with these instruction execution system, apparatus or device.

[0245] The above-described embodiments of the present application do not constitute a limitation on the protection scope of the present application.

Claims

1. A method for detecting slippage of a handling device, characterized in that: Applied to a transporting device, the transporting device is provided with an image acquisition device and an odometer, and the method includes: During the movement of the transport equipment along the travel path, first detection data collected by the image acquisition device and second detection data collected by the odometer are acquired; wherein the first detection data includes a plurality of initial image frames; Determining a target image frame that meets a preset condition from among the multiple initial image frames, and determining a first position of a first feature point in the first image frame, and a second position of the first feature point in a second image frame adjacent to the first image frame; wherein the target image frame includes the first image frame and the second image frame; Determining a first detection distance corresponding to the image acquisition device based on the first position and the second position, and determining a second detection distance corresponding to the odometer based on the second detection data; A slip condition of the transport equipment is determined based on the first detection distance and the second detection distance.

2. The method according to claim 1, characterized in that The determining of a target image frame satisfying a preset condition from among the multiple initial image frames comprises: Determining the brightness of each of the initial image frames and the number of feature points on each of the initial image frames; wherein the travel path of the transport equipment includes a track, and the feature points are used to represent identification information of the track; Among the multiple initial image frames, the initial image frames whose brightness is within a preset brightness value range and / or whose number of feature points is greater than a first preset number threshold are determined as target image frames that meet the preset conditions.

3. The method according to claim 1, characterized in that The determining of a first position of the first feature point in the first image frame, and the determination of a second position of the first feature point in a second image frame adjacent to the first image frame, the method further comprising: determining a plurality of first initial feature points in the first image frame and a plurality of second initial feature points in the second image frame; At least one target feature point is determined based on the first initial feature point and the second initial feature point; wherein the at least one target feature point is located in both the first image frame and the second image frame, and the at least one target feature point includes the first feature point.

4. The method according to claim 3, characterized in that The determining of at least one target feature point based on the first initial feature point and the second initial feature point includes: Determining, based on the first initial feature points and the second initial feature points, a plurality of first candidate feature points located simultaneously in the first image frame and the second image frame; Determining a first candidate position of each first candidate feature point among the plurality of first candidate feature points in the first image frame, and a second candidate position of each first candidate feature point in the second image frame; Determine the displacement difference between the first candidate position and the second candidate position corresponding to each first candidate feature point; Based on the displacement differences, the at least one target feature point is determined from the plurality of first candidate feature points.

5. The method according to claim 4, characterized in that The determining, based on each of the displacement differences, the at least one target feature point from the plurality of first candidate feature points comprises: A first candidate feature point among the plurality of first candidate feature points, the first candidate feature point whose displacement difference reaches a preset displacement difference threshold, is determined as the target feature point.

6. The method according to claim 5, characterized in that The method further comprises: Determine the number of the first candidate feature points and the number of the second candidate feature points; wherein the second candidate feature points are first candidate feature points whose displacement difference does not reach the preset displacement difference threshold among the multiple first candidate feature points; The proportion of abnormal feature points is determined based on the number of the first candidate feature points and the number of the second candidate feature points.

7. The method according to claim 6, characterized in that The determining the slippage condition of the transport equipment based on the first detection distance and the second detection distance includes: When the proportion of abnormal feature points is less than a preset ratio threshold, determining a first distance difference between the first detection distance and the second detection distance; If the first distance difference exceeds the target distance threshold, it is determined that the transport equipment has slipped; or, If the first distance difference exceeds a target distance threshold, and a duration in which the first distance difference exceeds the target distance threshold exceeds a target time threshold, it is determined that the transport equipment has slipped.

8. The method according to claim 7, characterized in that The transport equipment moves on the track via running wheels, and the method further comprises: Determining the friction coefficient between the running wheels of the transport device and the track, and the transport state of the transport device; wherein the transport state includes an empty state and a loaded state; The target distance threshold and the target time threshold are determined based on the friction coefficient and / or the transport state.

9. The method according to claim 6, characterized in that The method further comprises: When the proportion of the abnormal feature points is greater than or equal to a preset ratio threshold, the target image frame is discarded and the detection information of the image acquisition device is obtained again.

10. The method according to claim 3, characterized in that Before determining the plurality of first initial feature points in the first image frame and the plurality of second initial feature points in the second image frame, the method further includes: Performing block processing on a plurality of initial feature points of the target image frame to divide the target image frame into a plurality of grid units; A screening process is performed on a grid unit including a plurality of initial feature points so that the number of the initial feature points in the grid unit is less than or equal to a second preset number threshold.

11. The method according to claim 1, wherein The method further comprises: determining a number of feature points in a third image frame among the plurality of initial image frames; When the number of feature points in the third image frame is less than a third preset number threshold, determining whether the third image frame includes the first reference point; In a case where the third image frame includes the first reference point, if the image acquisition device acquires a fourth image frame including the second reference point, determining a third detection distance corresponding to the image acquisition device based on the first reference point and the second reference point, and determining a fourth detection distance corresponding to the odometer; A second distance difference between the third detection distance and the fourth detection distance is determined, and when the second distance difference exceeds a target distance threshold, it is determined that the transport equipment has slipped.

12. The method according to claim 11, characterized in that The transport equipment moves on the track through the running wheels. The track includes multiple sub-tracks. The reference point is used to represent the connection mark between two sub-tracks. The detection distance corresponding to two adjacent reference points is the length of the sub-track between the two adjacent reference points.

13. The method according to any one of claims 1 to 12, characterized in that The transport equipment includes a first running wheel and a second running wheel, and the image acquisition device includes a first optical flow sensor arranged between the two first running wheels, and a second optical flow sensor arranged between the two second running wheels; The step of obtaining the first detection data acquired by the image acquisition device during the movement of the transport equipment along the travel path includes: During the process of the transport device moving on the first track via the first traveling wheel, acquiring detection information corresponding to the first track collected by the first optical flow sensor; or, During the process of the transport device moving on the second track via the second running wheels, detection information corresponding to the second track collected by the second optical flow sensor is obtained.

14. The method according to any one of claims 1 to 12, characterized in that After determining the slippage condition of the handling equipment, the method further includes: When it is determined that the transport equipment has slipped, the output device is controlled to output an alarm message, and the position of the transport equipment is adjusted based on the first detection distance.

15. A device for detecting slippage of a handling device, characterized in that: include: an acquisition module configured to: acquire first detection data acquired by an image acquisition device of the transporting device and second detection data acquired by an odometer of the transporting device during movement of the transporting device along a driving path; wherein the first detection data includes a plurality of initial image frames; A first determining module is configured to: determine a target image frame that meets a preset condition from the multiple initial image frames, and determine a first position of a first feature point in the first image frame, and a second position of the first feature point in a second image frame adjacent to the first image frame; wherein the target image frame includes the first image frame and the second image frame; a second determining module configured to: determine a first detection distance corresponding to the image acquisition device based on the first position and the second position, and determine a second detection distance corresponding to the odometer based on the second detection data; The third determining module is configured to determine a slip condition of the transport equipment based on the first detection distance and the second detection distance.

16. A transport device, characterized in that: Including image acquisition device, odometer and control device; The image acquisition device is configured to: acquire first detection data during the movement of the transport device along the travel path; wherein the first detection data includes a plurality of initial image frames; The odometer is configured to: collect second detection data during the movement of the transport device along the driving path; The control device is configured to: obtain the first detection data and the second detection data; Determining a target image frame that meets a preset condition from among the multiple initial image frames, and determining a first position of a first feature point in the first image frame, and a second position of the first feature point in a second image frame adjacent to the first image frame; wherein the target image frame includes the first image frame and the second image frame; Determining a first detection distance corresponding to the image acquisition device based on the first position and the second position, and determining a second detection distance corresponding to the odometer based on the second detection data; A slip condition of the transport equipment is determined based on the first detection distance and the second detection distance.

17. An electronic device, characterized in that: include: one or more processors and memory; The memory is configured to: store one or more programs; Wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method for detecting slippage of a handling equipment according to any one of claims 1-14.

18. A computer-readable storage medium, characterized in that A computer program is stored thereon, and when the computer program is executed by a processor, the method for detecting slippage of a handling equipment according to any one of claims 1 to 14 is implemented.

19. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a processor, implements the method for detecting slippage of a handling equipment according to any one of claims 1 to 14.

Images