Object separation method, apparatus, electronic device, and machine-readable storage medium

Abstract

The application provides an object separation method and device, electronic equipment and machine readable storage medium. The method comprises the following steps: determining real-time position information of objects on a control module according to current frame acquisition data of an image acquisition device; determining actual position information of the objects according to stored historical position information of the objects and the real-time position information; determining sorting results of the objects according to stored historical sorting results of the objects and the actual position information; determining an optimal module speed according to the sorting results of the objects, and performing object separation processing according to the optimal module speed. The method can realize stable and uniform separation of mixed objects.

Description

Technical Field

[0001] This application relates to the field of logistics automation, and more particularly to a method, apparatus, electronic device, and machine-readable storage medium for separating items. Background Technology

[0002] Existing automated sorting systems can only operate normally and stably when each item enters the scanning range one by one. In order to realize automated sorting in the logistics field and replace manual sorting operations, items need to pass through the sorting area one by one, and a certain safe distance needs to be maintained between items.

[0003] To separate a group of mixed items and transform the mixed arrangement into individual items, allowing them to pass through the sorting area in an orderly and even manner, the individual item separation system was developed. An automated sorting system, in conjunction with the individual item separation system, enables fully automated package transport and sorting.

[0004] How to achieve stable and uniform separation of mixed objects has become a pressing technical problem in object separation solutions. Summary of the Invention

[0005] In view of this, this application provides a method, apparatus, electronic device, and machine-readable storage medium for separating objects.

[0006] According to a first aspect of the embodiments of this application, an object separation method is provided, comprising:

[0007] Based on the current frame data acquired by the image acquisition device, determine the real-time position information of the object on the control module;

[0008] Based on the stored historical location information of the object and the real-time location information, the actual location information of the object is determined;

[0009] Based on the stored historical sorting results of the objects and the actual location information, the sorting result of each object is determined;

[0010] Based on the sorting results of the objects, the optimal module speed is determined, and the objects are separated according to the optimal module speed.

[0011] According to a second aspect of the embodiments of this application, an object separation device is provided, comprising:

[0012] The first determining unit is used to determine the real-time position information of the object on the control module based on the current frame data acquired by the image acquisition device.

[0013] The second determining unit is used to determine the actual location information of the object based on the stored historical location information of the object and the real-time location information.

[0014] The third determining unit is used to determine the sorting result of each object based on the stored historical sorting results of the objects and the actual location information.

[0015] The separation processing unit is used to determine the optimal module speed based on the sorting results of the objects, and to perform object separation processing based on the optimal module speed.

[0016] According to a third aspect of the present application, an electronic device is provided, including a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions executable by the processor, the processor being configured to execute the machine-executable instructions to implement the method provided in the first aspect.

[0017] According to a fourth aspect of the embodiments of this application, a machine-readable storage medium is provided, wherein machine-executable instructions are stored therein, and when the machine-executable instructions are executed by a processor, the method provided in the first aspect is implemented.

[0018] The object separation method of this application improves the accuracy of the determined object position information by referring to the real-time position information and historical position information of the object when determining the position information of the object on the control module, and determining the actual position information based on the real-time position information and historical position information of the object. Furthermore, based on the stored historical sorting results of the object and the actual position information of the object, the sorting result of the object is determined, and the optimal module speed is determined based on the determined sorting result. The object separation process is performed based on the optimal module speed, which improves the reliability of the determined module speed and effectively realizes the stable and uniform separation of mixed objects. Attached Figure Description

[0019] Figure 1 This is a schematic flowchart of an object separation method provided in an embodiment of this application;

[0020] Figure 2A This is a schematic diagram of the architecture of an object separation system provided in an embodiment of this application;

[0021] Figure 2B This is a schematic diagram of the structure of an object separation system provided in an embodiment of this application;

[0022] Figure 2C This is a schematic diagram of an object separation scenario provided in an embodiment of this application;

[0023] Figure 2D This is a schematic diagram of an object separation scenario provided in an embodiment of this application;

[0024] Figure 3This is a schematic diagram of an object separation process provided in an embodiment of this application;

[0025] Figure 4 This is a schematic diagram of the structure of an object separation device provided in an embodiment of this application;

[0026] Figure 5 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0027] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0028] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0029] To enable those skilled in the art to better understand the technical solutions provided in the embodiments of this application, the architecture of the separation system will be briefly described below.

[0030] like Figure 2A As shown, the object separation system mainly includes an object separation device and a vision recognition system. The vision recognition system's field of view covers the working area of ​​the object separation device, can capture the real-time position of the objects, and calculates the optimal separation sequence and path through data algorithm analysis. Finally, the object separation device realizes the separation, pulling, and queuing of the objects.

[0031] The object separation equipment includes multiple control modules, the speed of which can be controlled independently. This allows the objects within the working area of ​​the object separation equipment to move at different speeds along the conveying direction, thereby achieving object separation, stretching, and queuing.

[0032] For example, common object separation equipment may include a belt separator, which uses a belt (or conveyor belt) as a control module. By controlling the speed of different conveyor belts, the objects are controlled to move at different speeds along the conveying direction, thereby achieving object separation, stretching, and queuing.

[0033] It should be noted that the mixed stacked objects are placed at the input end of the object separation device, and the objects can be manually or automatically placed into the working area of ​​the object separation device.

[0034] Items separated by the item separation system can then enter the sorting area for sorting, or proceed to other subsequent processing steps.

[0035] To make the above-mentioned objectives, features and advantages of the embodiments of this application more apparent and understandable, the technical solutions in the embodiments of this application will be further described in detail below with reference to the accompanying drawings.

[0036] It should be noted that the sequence number of each step in the embodiments of this application does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0037] Please see Figure 1 This is a flowchart illustrating an object separation method provided in an embodiment of this application, as shown below. Figure 1 As shown, the object separation method may include the following steps:

[0038] Step S100: Determine the real-time position information of the object on the control module based on the current frame acquisition data of the image acquisition device.

[0039] Step S110: Determine the actual location information of the object based on the stored historical location information and the real-time location information of the object.

[0040] In this embodiment of the application, in order to achieve object separation, image data on the control module can be acquired by an image acquisition device, and the position information of the object on the control module can be determined based on the acquired image data.

[0041] Considering that the position information of objects determined by object detection based on single-frame image data may contain detection noise (such as missed detections or false detections), when determining the position information of objects, we can refer not only to the detection results of the most recently acquired image data, but also to the detection results of the image data of historical frames, so as to improve the accuracy of the determined position information of objects.

[0042] Accordingly, in order to achieve object separation, the real-time position information of the object on the control module can be determined based on the latest acquired data of the image acquisition device (which can be called the current frame acquisition data), and the actual position information of the object can be determined based on the stored historical position information of the object and the real-time position information.

[0043] The real-time position information of the object is determined based on the current frame data acquired by the image acquisition device, meaning the real-time position information is determined based on the latest single-frame acquisition data. The actual position information of the object is determined based on the object's historical position information and real-time position information, meaning the actual position information of the object is determined based on multiple frames of acquisition data to reduce the impact of detection noise in single-frame acquisition data on the accuracy of position information determination. The historical position information of the object refers to the actual position information of the object determined before the current moment.

[0044] Specifically, for the data acquired in the current frame, the historical location information of the object can be the actual location information of the object determined according to the scheme provided in the embodiments of this application based on the data acquired in the previous frame.

[0045] For example, assuming the current frame data acquisition time is t2 and the previous frame data acquisition time is t1, then for time t2, the historical position information of the object can be the actual position information of the object determined at time t1.

[0046] It should be noted that for real-time location information determined based on the first frame of data acquired by the image acquisition device, the historical location information of the object can be empty; for real-time location determined based on data acquired by the image acquisition device other than the first frame, the historical location information of the object can be determined based on the historical frame data acquired by the image acquisition device.

[0047] Step S120: Determine the sorting result of each object based on the stored historical sorting results and the actual location information of the objects.

[0048] In this embodiment of the application, the stored historical information of the objects may include not only the historical location information mentioned above, but also the historical sorting results, that is, the sorting relationship of each object determined in the previous frame data processing.

[0049] For example, the sorting result refers to the result of sorting each object to be separated according to its position on the control module from front to back or from back to front (usually from front to back).

[0050] Once the actual position information of the objects on the control module is determined in the manner described above, the calculation of the current frame sorting result can be guided by the stored historical sorting results, thereby improving the reliability of the current frame sorting result and making the object separation more stable.

[0051] For example, the sorting result of each object can be determined based on the historical sorting results of the stored objects and the actual position information of each object during the current frame data processing determined in the manner described above.

[0052] For example, items that appear earlier in the sorting list are separated first.

[0053] For example, considering that in real-world scenarios, there may be multiple objects on the control module whose foremost positions are relatively close, in which case it may not be possible to directly determine the sorting result of each object based on its position. In this case, there may be multiple possible sorting results.

[0054] Accordingly, when determining the sorting result of each object based on the stored historical sorting results and the actual location information of the objects, the determined sorting result may include multiple candidate sorting results.

[0055] It should be noted that the number of candidate sorting results can also be only one. In this case, the sorting result can be used for separation control.

[0056] Step S130: Determine the optimal module speed based on the sorting results of each object, and perform object separation processing based on the optimal module speed.

[0057] In this embodiment of the application, after the sorting result of each object is determined in the manner described above, the optimal module speed can be determined based on the determined sorting result, and the object separation process can be performed based on the optimal module speed.

[0058] It should be noted that when the determined sorting result includes multiple candidate sorting results, the optimal module speed under each candidate sorting result can be determined separately, and the final optimal module speed can be determined from them.

[0059] For example, for any control module covered by any object, the speed of the control module is controlled to be consistent with the determined optimal module speed.

[0060] It can be seen that, in Figure 1 In the method flow shown, when determining the position information of objects on the control module, the real-time position information and historical position information of the objects are referenced. Based on the real-time position information and historical position information of the objects, the actual position information is determined, which improves the accuracy of the determined position information of the objects. Furthermore, based on the stored historical sorting results of the objects and the actual position information of the objects, the sorting result of the objects is determined, and the optimal module speed is determined based on the determined sorting result. The object separation process is performed based on the optimal module speed, which improves the reliability of the determined module speed and effectively realizes the stable and uniform separation of mixed objects.

[0061] In some embodiments, determining the actual location information of an object based on stored historical location information and real-time location information may include:

[0062] For any object, if there is historical location information for the object and real-time location information for the object, the real-time location information of the object is updated based on the historical location information to determine the actual location information of the object.

[0063] If historical location information for an object exists, but real-time location information for the object does not exist, the actual location information of the object shall be determined based on the historical location information of the object.

[0064] If there is no historical location information for the object, but there is real-time location information for the object, the actual location information of the object is determined based on the real-time location information.

[0065] For example, objects detected based on data acquired by an image acquisition device can be tracked in order to correlate information about the objects in the temporal domain, i.e., to identify the same object in different frames.

[0066] For example, in the process of determining the actual location information of an object based on the stored historical location information and real-time location information, for any object (an object with historical location information or an object with real-time location information), it can be determined whether the object has historical location information and whether the object has real-time location information.

[0067] For objects that have both historical and real-time location information, their real-time location information can be updated based on their historical location information to determine their actual location.

[0068] For example, based on the object's historical location information and its moving speed, the object's position in the current frame can be predicted to obtain predicted position information. The predicted position information and the aforementioned real-time position information can then be fused, such as by calculating a weighted average, to obtain the object's actual position information.

[0069] For example, suppose object A moves at a speed of v. a Historical location is x a1 (Assuming the object's movement direction is the positive x-axis), and the frame rate is FR, then the predicted position information x is obtained based on the object's movement speed and historical position information. a2 It can be x a2 =x a1 +v a / FR.

[0070] The movement speed of the object can be determined based on the speed of the control module covered by the object.

[0071] For an object covering a single control module, its moving speed can be the speed of that control module; for an object covering multiple control modules, its moving speed can be the weighted average of the speeds of those multiple control modules.

[0072] For example, assuming that object A covers modules 1, 2 and 3, and the area of ​​object A on modules 1, 2 and 3 accounts for 20%, 70% and 10% respectively, then the weighting weights of the speeds of each module can be 0.2, 0.7 and 0.1 respectively.

[0073] In the case of predicting the position of objects in the current frame, the area covered by the objects on each module can be determined based on the detection results of the previous frame image.

[0074] For objects that have historical location information but no real-time location information, such as objects that have left the control module in the current frame, or objects that were missed in the current frame, the actual location information of the object can be determined based on the object's historical location information.

[0075] For example, based on the object's historical location information and its moving speed, the object's position in the current frame can be predicted to obtain predicted position information, which can then be used as the object's actual position information.

[0076] It can be seen that the actual position information of objects that have left the control module in the current frame can also be obtained. Therefore, when controlling the speed of the first object in the control module, it is necessary to ensure that the distance between it and the previous object that left the control module meets the separation distance requirement.

[0077] For objects that do not have historical location information but have real-time location information, i.e., objects that enter the control module in the current frame, the actual location information of the object can be determined based on the object's real-time location information. For example, the object's real-time location information can be determined as the object's actual location information.

[0078] It should be noted that, once the actual position information of an object is determined in the manner described above, the determined actual position information of the object can be stored as the latest historical position information. That is, the actual position information of each object in the current frame data processing process determined in the manner described above can be used as the historical position information of the object in the next frame data processing process.

[0079] In some embodiments, determining the sorting result of each object based on the stored historical sorting results and the actual location information may include:

[0080] For any two objects, based on the historical sorting results of the two objects, add a first offset to the actual position information of the object that is sorted first, and add a second offset to the actual position information of the object that is sorted second. Based on the updated actual position information of the two objects, determine the sorting result of the two objects. The first offset is greater than the second offset, and the larger the offset, the greater the forward shift of the position.

[0081] For example, considering that the sorting result of objects on the control module will not change under normal circumstances during object movement, and that objects sorted first will be separated first, in order to improve the stability of object separation, in the process of determining the sorting result of the current frame, for any two objects, a first offset can be added to the actual position information of the object sorted first among the two objects based on the historical sorting results of the two objects, and a second offset can be added to the actual position information of the object sorted second among the two objects, so as to obtain the updated actual position information of the two objects, and the sorting result of the two objects can be determined based on the updated actual position information of the two objects, so as to correct the detection error of single frame data and ensure that the sorting result of objects in the current frame is as consistent as possible with the historical sorting result.

[0082] For example, assuming that object A is before object B in the historical sorting results, the actual position information of object A and object B determined in accordance with the method described in the above embodiments (let's call them Loc A and Loc B respectively) can be updated by adding an offset based on the historical sorting results to obtain the updated actual position information: Loc A+off1 (i.e., the first offset mentioned above) and Loc B+off2 (i.e., the second offset mentioned above).

[0083] For example, the first offset is greater than the second offset, and the larger the offset, the greater the forward offset. That is, for objects that are ranked earlier in the history, a larger forward offset is added to them.

[0084] It should be noted that the second offset mentioned above can be 0, meaning that for objects that are sorted later, no forward offset needs to be added.

[0085] In some embodiments, determining the sorting result of each object based on the stored historical sorting results and the actual location information may include:

[0086] For any two objects, if the maximum area of ​​the overlapping region of the outlines of the two objects exceeds a preset area threshold during the forward movement of the other object, then the other object is determined to be ranked after the ranking of the first object.

[0087] For example, to avoid collisions during object separation, for any two objects, based on their actual position information, while fixing the position of one object, the trajectory of the other object moving forward can be simulated. The maximum area of ​​the overlapping region of the two objects' outlines during this forward movement is determined, and compared to a preset area threshold. If the maximum area exceeds the preset threshold, it is determined that the other object will collide with the first object during its forward movement. In this case, the order of the other object can be determined to be after the order of the first object to avoid collisions during object separation.

[0088] For example, such as Figure 2C As shown, assuming that for objects A and B, with the position of object A (one of the aforementioned objects) fixed, during the forward movement of object B (the other of the aforementioned objects), if the maximum area of ​​the overlapping region of the contours of object A and object B exceeds a preset area threshold, it can be determined that object B will collide with object A during its forward movement. In this case, it can be determined that object B is ordered after object A, that is, object A is separated first, and object B is separated later.

[0089] It should be noted that, in the embodiments of this application, the sorting results and module speed determined in the above manner are both determined based on the acquisition data of the current frame. When new acquisition data is obtained through the image acquisition device, the sorting results and module speed of the objects can be updated.

[0090] In some embodiments, determining the candidate sorting result for each object based on the stored historical sorting results and the actual location information may include:

[0091] If the distance between the frontmost edges of multiple objects is less than a preset distance threshold, the sorting results of these multiple objects will be used as candidate sorting results for these multiple objects.

[0092] For example, considering that the accuracy of recognizing the position information of objects and the accuracy of controlling the speed of objects are both limited in real-world scenarios, and that the positional relationship of objects may change slightly due to some reasons during movement, the order of multiple objects that are close to each other at the foremost point cannot be directly determined based on their positional relationship.

[0093] Accordingly, if the distance between the frontmost edges of multiple objects is less than a preset distance threshold, the sorting results of these multiple objects can all be used as candidate sorting results for these multiple objects.

[0094] For example, such as Figure 2DAs shown, taking the positive x-axis direction along the object's movement direction as an example, assuming that the difference between the x-coordinates of the frontmost points of objects A, B, and C is less than a preset threshold (i.e., the distance between the frontmost points is less than a preset distance threshold), then for objects A, B, and C, the candidate sorting results can include ABC, ACB, BAC, BCA, CAB, and CBA.

[0095] In some embodiments, determining the optimal module speed based on the sorting results of the objects may include:

[0096] If there is only one sorting result, the module speed with the lowest cost under that sorting result is determined as the optimal module speed.

[0097] For example, if the number of sorting results of objects determined in step S120 is 1, then the sorting result can be determined to be the optimal sorting result.

[0098] Based on a preset cost function, the module speed with the lowest cost under the sorting result can be determined, and this module speed can be determined as the optimal module speed.

[0099] For example, for any module speed under any sorting result, the cost can be determined based on the distance between the separated objects under that module speed, the consistency of the speed of different modules covered by the same object, and the stability of the speed between objects. The specific implementation can be explained in the following with specific examples.

[0100] In some embodiments, determining the optimal module speed based on the sorting results of each object, and performing object separation processing based on the optimal module speed, may include:

[0101] When there are multiple sorting results, determine the target module speed with the lowest cost for each sorting result;

[0102] Based on the cost of the target module speed, the target module speed with the lowest cost is determined as the optimal module speed.

[0103] For example, the number of sorting results of objects determined in step S120 is multiple. That is, when there are multiple candidate sorting results, the module speed with the lowest cost under each candidate sorting result (which can be called the target module speed) can be determined separately.

[0104] Based on the cost of the target module speed, the target module speed with the lowest cost can be determined as the optimal module speed.

[0105] For example, for any module speed under any sorting result, the cost can be determined based on the distance between the separated objects under that module speed, the consistency of the speed of different modules covered by the same object, and the stability of the speed between objects. The specific implementation can be explained in the following with specific examples.

[0106] In one example, for any module speed under any sorting result, the cost of that module speed under that sorting result is determined based on a first type of cost, a second type of cost, and a third type of cost;

[0107] At this module speed, when the last part of the first sorted object in the sorting result passes the separation line, the closer the distance between the second-first sorted object and the separation line is to the separation distance, the smaller the first type of cost.

[0108] At this module speed, for any object, the closer the speed of each module covered by the object is to the speed of the object, the smaller the second type of cost;

[0109] At this module speed, for any object, if the object's speed is greater than or equal to its speed in the previous frame, the third type cost is 0; if the object's speed is less than its speed in the previous frame, the greater the absolute value of the difference between the object's speed and its speed in the previous frame, the greater the third type cost.

[0110] For example, the specific methods for determining the first type of cost, the second type of cost, and the third type of cost can be explained in the following text with specific examples.

[0111] To enable those skilled in the art to better understand the technical solutions provided in the embodiments of this application, the technical solutions provided in the embodiments of this application are described below with reference to specific examples.

[0112] In this embodiment, in order to achieve a uniform and stable separation effect, an object separation scheme based on temporal correlation and module units is provided.

[0113] By associating the detection results of historical frames with those of the current frame, objects are tracked and predicted across the entire control module, establishing a complete, accurate, and stable sorting relationship. Furthermore, using the speed of all modules as the functional unit, an objective function for the separation system is constructed to control and separate objects at the lowest cost, thereby improving the smoothness of the system.

[0114] For example, based on the state of all objects and modules in historical frames, all detected objects can be tracked in the current frame to obtain historical information about the objects and guide their sorting, making the sorting result consistent with the historical frames, and multiple candidate sorting results can be retained. Then, based on the historical frames, missing objects are predicted, and those within the predicted area are added to the sorting results to provide supervision information for the first object at the exit. Next, based on the sorting results of all objects, using the speed of all modules as the functional unit, and the separation distance of all objects, the uniformity of the speed of each part of the object, and the degree of change in module speed as the functional objectives, the appropriate module speed for each sorting result is calculated. Finally, the module speed with the lowest objective function cost is selected from the candidate sorting results and sent to the control unit for object separation control. A schematic diagram of this structure can be shown below. Figure 2B As shown.

[0115] The following explains some implementation details.

[0116] 1. Obtain the position information of objects: Obtain the position information of all objects on the control module through the camera (i.e. the image acquisition device mentioned above).

[0117] 2. Temporal association: Information on all objects in the stored historical frames (such as location information (i.e., the historical location information mentioned above), sorting results (i.e., the historical sorting results mentioned above), and module speed, etc.) are used to associate the objects detected in the current frame with tracking algorithms (such as Kalman filtering algorithm and Hungarian algorithm).

[0118] For example, the association results can include the following three cases:

[0119] 2.1 Successful association (i.e., real-time location information exists, and historical location information exists): Update the object's location information, predict the location information based on the location information of historical frames, and fuse the predicted location information with the real-time location information to obtain the actual location information;

[0120] 2.2. Location information exists in historical frames but is not successfully associated (i.e., no real-time location information exists): Based on the location information of historical frames, prediction is made to obtain predicted location information, which is then used as the actual location information.

[0121] 2.3. For locations that exist in the current frame but have not been successfully associated (i.e., historical location information does not exist): Use the information of the current frame (i.e., real-time location information) to determine the real-time location information as the actual location information.

[0122] 3. Calculate the object separation order: Based on the historical sorting results, guide the calculation of the current frame sorting results, thereby making the separation more stable.

[0123] For example, the calculation of the sorting results can take the following factors into account:

[0124] 3.1 Sort according to historical sorting results: Based on the historical sorting results, calculate the order of priority between each pair of objects, and add a forward offset (e.g., 50mm) to the objects that are ranked earlier, thereby determining the separation order of the objects;

[0125] 3.2 Collision Relationship between Objects: By fixing one object and moving another object forward, the maximum area of ​​the overlapping region between the outlines of the two objects is determined. If the maximum area exceeds a preset area threshold, it is determined that the two objects will collide, and the collided object (i.e., one of the objects in the above embodiment, i.e., the object at the fixed position) will be sorted in front of the other object.

[0126] 3.3 For combinations with ambiguous sorting results, retain multiple sorting results: If the distance between the frontmost edges of multiple objects is less than a preset distance threshold, it is considered that there are multiple possibilities for the sorting result, and all multiple sorting results are retained.

[0127] 4. Establish the target separation function: Based on the sorting results of all objects, using the velocity v of all modules as the function unit, and taking the separation distance dist of all objects, the uniformity of the velocity V of each part of the object, and the degree of variation of the module velocity as the function objective, calculate the appropriate module velocity v under each sorting result. 1～n The target separation optimization function is shown in Equations 1-1 and 1-2, which enables the module to separate the packages at a uniform spacing within the allowable speed.

[0128] Where V0 and V1 represent the velocities of the object to be separated (0) and the object to be separated (1) next, respectively, and t x x represents the distance from the separation line (the point where the objects are separated). 0_min x represents the minimum distance of object 0 (the position of the last segment of object 0). 1_max This represents the maximum distance of object 1 (the position of the foremost part of object 1), where the velocity V of the object is as shown in Equation 1-2, derived from the velocities v of each module. i Combined with weight w i get.

[0129] s1: V1*(t x -x 0_min )=V0*(t x -dist-x 1_max ) 1-1

[0130] V = sum i＝1～n (w i *v i ) 1-2

[0131] The optimization objective of s1 is to make the distance between object 1 and object 0 (the distance between the front end of object 1 and the back end of object 0) as close as possible to the preset separation distance (i.e., dist) when object 0 reaches the separation line. The cost of different module speeds is determined based on the difference between the distance between object 1 and object 0 (the distance between the front end of object 1 and the back end of object 0) and the preset separation distance when object 0 reaches the separation line. The greater the difference, the greater the cost (the value of s1).

[0132] For example, for any object that covers multiple modules, the speed of the object can be a weighted average of the speeds of the covered modules.

[0133] For example, the weighted weights of the speeds of each module (i.e., w) i It can be determined based on the area ratio of the object on each module.

[0134] For example, assuming that object A covers modules 1, 2 and 3, and the area of ​​object A on modules 1, 2 and 3 accounts for 20%, 70% and 10% respectively, then the weighting weights of the speeds of each module can be 0.2, 0.7 and 0.1 respectively.

[0135] The penalty term of the objective function can include the following two:

[0136] 4.1 As shown in Equation 1-3, this means that the speed of all modules corresponding to each object (i.e., all modules covered by the object) tends to be consistent. For any object, the greater the difference between the speed of each module in all modules covered by the object and the speed of the object, the greater the cost (value of s2).

[0137] s2:v 1～n *(t x -dist-x 1_max )=V1*(t x -dist-x 1_max ) 1-3

[0138] 4.2 As shown in Equations 1-4, in order to keep the system smooth, the speed of objects should maintain an increasing trend. That is, for any object, the speed of the current frame (e.g., V1) needs to be greater than or equal to the speed of the previous frame (e.g., V′1). If the speed of the object in the current frame is less than the speed of the previous frame, the greater the speed difference, the greater the cost (the value of s3).

[0139] s3:V′1*(t x -dist-x 1_max )≤V1*(t x -dist-x 1_max ) 1-4

[0140] 5. Optimize module speed: Based on the current sorting results, optimize the target separation function to obtain the optimized module speed. This optimization process includes two cases:

[0141] 5.1 The number of sorting results is 1: Assign weights (empirical values, set according to actual needs, the sum of weights is 1) to Equations 1-1, 1-3 and 1-4 and add them together to generate the cost function (also called loss function) shown in Equation 1-5. Optimize the cost function s by using traversal and other methods to obtain the module speed with the minimum cost as the optimal module speed.

[0142] s = w1*s1 + w2*s2 + w3*s3 (1-5)

[0143] 5.2 The number of sorting results is multiple, that is, there are multiple candidate sorting results: track all candidate sorting results, calculate the corresponding module speed and cost function for each group of object separation order, obtain the module speed with the minimum cost under each candidate sorting result (i.e. the target module speed mentioned above), and select the target module speed with the minimum cost as the optimal module speed.

[0144] 6. Perform separation function: Send the optimal module speed to the control system to separate the object.

[0145] For example, the schematic diagram of the object separation process in this embodiment can be as follows: Figure 3 As shown, it includes the following steps:

[0146] 1. Obtain the position information of all objects on the control module (i.e., the real-time position information mentioned above) through the data collected by the camera;

[0147] 2. Based on information from historical frames, track the detection results of the current frame and correlate the information of objects in the temporal domain;

[0148] 3. For objects that exist in historical frames but are not detected in the current frame, make predictions and retain the objects within the prediction area;

[0149] 4. Combine tracking and prediction information to calculate complete object information on the system;

[0150] 5. Using the sorting results in the historical frames as a guide, sort the complete objects and retain all candidate sorting results;

[0151] 6. For each sorting result, establish a function objective with the speed of all modules as the function unit;

[0152] 7. Calculate the module speed that minimizes the objective function cost under the current sorting results (i.e., the target module speed mentioned above);

[0153] 8. Select the optimal module speed from all sorting results and use this information to update the historical frames;

[0154] 9. Send the optimal module speed to the control unit so that it can perform the separation function on the object.

[0155] The method provided in this application has been described above. The apparatus provided in this application is described below:

[0156] Please see Figure 4 This is a schematic diagram of the structure of an object separation device provided in an embodiment of this application, as shown below. Figure 4 As shown, the object separation device may include:

[0157] The first determining unit 410 is used to determine the real-time position information of the object on the control module based on the current frame acquisition data of the image acquisition device.

[0158] The second determining unit 420 is used to determine the actual location information of the object based on the stored historical location information of the object and the real-time location information.

[0159] The third determining unit 430 is used to determine the sorting result of each object based on the stored historical sorting results of the objects and the actual location information.

[0160] The separation processing unit 440 is used to determine the optimal module speed based on the sorting result of each object, and to perform object separation processing based on the optimal module speed.

[0161] In some embodiments, the second determining unit 420 determines the actual location information of the object based on the stored historical location information of the object and the real-time location information, including:

[0162] For any object, if there is historical location information for the object and real-time location information for the object, the real-time location information of the object is updated based on the historical location information to determine the actual location information of the object.

[0163] If historical location information for an object exists, but real-time location information for the object does not exist, the actual location information of the object shall be determined based on the historical location information of the object.

[0164] If there is no historical location information for the object, but there is real-time location information for the object, the actual location information of the object is determined based on the real-time location information.

[0165] In some embodiments, the third determining unit 430 determines the sorting result of each object based on the stored historical sorting results of the objects and the actual location information, including:

[0166] For any two objects, based on the historical sorting results of the two objects, add a first offset to the actual position information of the object that is sorted first, and add a second offset to the actual position information of the object that is sorted second. Based on the updated actual position information of the two objects, determine the sorting result of the two objects. The first offset is greater than the second offset, and the larger the offset, the greater the forward shift of the position.

[0167] In some embodiments, the third determining unit 430 determines the sorting result of each object based on the stored historical sorting results of the objects and the actual location information, including:

[0168] For any two objects, if the maximum area of ​​the overlapping region of the outlines of the two objects exceeds a preset area threshold during the forward movement of the other object, then the other object is determined to be ranked after the ranking of the first object.

[0169] In some embodiments, the third determining unit 430 determines the sorting result of each object based on the stored historical sorting results of the objects and the actual location information, including:

[0170] If the distance between the frontmost edges of multiple objects is less than a preset distance threshold, the sorting results of these multiple objects will be used as candidate sorting results for these multiple objects.

[0171] In some embodiments, the separation processing unit 440 determines the optimal module speed based on the sorting result of the objects, including:

[0172] If there is only one sorting result, the module speed with the lowest cost under that sorting result is determined as the optimal module speed.

[0173] In some embodiments, the separation processing unit 440 determines the optimal module speed based on the sorting result of the objects, and performs object separation processing based on the optimal module speed, including:

[0174] When there are multiple sorting results, determine the target module speed with the lowest cost for each candidate sorting result.

[0175] Based on the cost of the target module speed, the target module speed with the lowest cost is determined as the optimal module speed.

[0176] In some embodiments, for any module speed under any sorting result, the cost of the module speed under that sorting result is determined based on a first type of cost, a second type of cost, and a third type of cost;

[0177] At this module speed, when the last part of the first sorted object in the sorting result passes the separation line, the closer the distance between the second-first sorted object and the separation line is to the separation distance, the smaller the first type of cost.

[0178] At this module speed, for any object, the closer the speed of each module covered by the object is to the speed of the object, the smaller the second type of cost;

[0179] At this module speed, for any object, if the object's speed is greater than or equal to its speed in the previous frame, the third type cost is 0; if the object's speed is less than its speed in the previous frame, the greater the absolute value of the difference between the object's speed and its speed in the previous frame, the greater the third type cost.

[0180] In some embodiments, the separation processing unit 440 performs object separation processing based on the optimal module speed, including:

[0181] For any control module covered by any object, the speed of the control module is controlled to be consistent with the determined optimal module speed.

[0182] This application provides an electronic device including a processor and a memory, wherein the memory stores machine-executable instructions that can be executed by the processor, and the processor executes the machine-executable instructions to implement the object separation method described above.

[0183] Please see Figure 5 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. The electronic device may include a processor 501 and a memory 502 storing machine-executable instructions. The processor 501 and the memory 502 can communicate via a system bus 503. Furthermore, by reading and executing the machine-executable instructions corresponding to the object separation logic in the memory 502, the processor 501 can execute the object separation method described above.

[0184] The memory 502 mentioned in this document can be any electronic, magnetic, optical, or other physical storage device that can contain or store information such as executable instructions, data, etc. For example, machine-readable storage media can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), solid-state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or combinations thereof.

[0185] In some embodiments, a machine-readable storage medium, such as Figure 5 The memory 502 in the machine-readable storage medium stores machine-executable instructions, which, when executed by a processor, implement the object separation method described above. For example, the storage medium may be ROM, RAM, CD-ROM, magnetic tape, floppy disk, or optical data storage device.

[0186] It should be noted that, in this document, relational terms such as "objective" and "target" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0187] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for separating objects, characterized in that, include: Based on the current frame data acquired by the image acquisition device, determine the real-time position information of the object on the control module; Based on the stored historical location information of the object and the real-time location information, the actual location information of the object is determined; Based on the stored historical sorting results of the objects and the actual location information, the sorting result of each object is determined; Based on the sorting results of the objects, the optimal module speed is determined, and the objects are separated according to the optimal module speed. Wherein, for any module speed under any sorting result, the cost of the module speed under that sorting result is determined based on the first type of cost, the second type of cost, and the third type of cost; At this module speed, when the last part of the first sorted object in the sorting result passes the separation line, the closer the distance between the second-first sorted object and the separation line is to the separation distance, the smaller the first type of cost. At this module speed, for any object, the closer the speed of each module covered by the object is to the speed of the object, the smaller the second type of cost; At this module speed, for any object, if the object's speed is greater than or equal to the object's speed in the previous frame, the third type cost is 0; if the object's speed is less than the object's speed in the previous frame, the greater the absolute value of the difference between the object's speed and the object's speed in the previous frame, the greater the third type cost. The optimal module speed is the module speed with the lowest cost.

2. The method according to claim 1, characterized in that, Determining the actual location information of an object based on its stored historical location information and its real-time location information includes: For any object, if there is historical location information for the object and real-time location information for the object, the real-time location information of the object is updated based on the historical location information to determine the actual location information of the object. If historical location information for an object exists, but real-time location information for the object does not exist, the actual location information of the object shall be determined based on the historical location information of the object. If there is no historical location information for the object, but there is real-time location information for the object, the actual location information of the object is determined based on the real-time location information.

3. The method according to claim 1, characterized in that, The process of determining the sorting result of each object based on the stored historical sorting results and the actual location information includes: For any two objects, based on the historical sorting results of the two objects, add a first offset to the actual position information of the object that is sorted first, and add a second offset to the actual position information of the object that is sorted second. Based on the updated actual position information of the two objects, determine the sorting result of the two objects. The first offset is greater than the second offset, and the larger the offset, the greater the forward shift of the position.

4. The method according to claim 1, characterized in that, The process of determining the sorting result of each object based on the stored historical sorting results and the actual location information includes: For any two objects, if the maximum area of ​​the overlapping region of the outlines of the two objects exceeds a preset area threshold during the forward movement of the other object, then the other object is determined to be ranked after the ranking of the first object.

5. The method according to claim 1, characterized in that, The process of determining the sorting result of each object based on the stored historical sorting results and the actual location information includes: If the distance between the frontmost edges of multiple objects is less than a preset distance threshold, the sorting results of these multiple objects will be used as candidate sorting results for these multiple objects.

6. The method according to claim 1, characterized in that, Determining the optimal module speed based on the sorting results of the items includes: If there is only one sorting result, the module speed with the lowest cost under that sorting result is determined as the optimal module speed.

7. The method according to claim 1, characterized in that, The step of determining the optimal module speed based on the sorting results of the objects, and performing object separation processing based on the optimal module speed, includes: When there are multiple sorting results, determine the target module speed with the lowest cost for each sorting result; Based on the cost of the target module speed, the target module speed with the lowest cost is determined as the optimal module speed.

8. The method according to claim 1, characterized in that, The object separation process based on the optimal module speed includes: For any control module covered by any object, the speed of the control module is controlled to be consistent with the determined optimal module speed.

9. An object separation device, characterized in that, include: The first determining unit is used to determine the real-time position information of the object on the control module based on the current frame data acquired by the image acquisition device. The second determining unit is used to determine the actual location information of the object based on the stored historical location information of the object and the real-time location information. The third determining unit is used to determine the sorting result of each object based on the stored historical sorting results of the objects and the actual location information. The separation processing unit is used to determine the optimal module speed based on the sorting result of each object, and to perform object separation processing based on the optimal module speed. Wherein, for any module speed under any sorting result, the cost of the module speed under that sorting result is determined based on the first type of cost, the second type of cost, and the third type of cost; At this module speed, when the last part of the first sorted object in the sorting result passes the separation line, the closer the distance between the second-first sorted object and the separation line is to the separation distance, the smaller the first type of cost. At this module speed, for any object, the closer the speed of each module covered by the object is to the speed of the object, the smaller the second type of cost; At this module speed, for any object, if the object's speed is greater than or equal to the object's speed in the previous frame, the third type cost is 0; if the object's speed is less than the object's speed in the previous frame, the greater the absolute value of the difference between the object's speed and the object's speed in the previous frame, the greater the third type cost. The optimal module speed is the module speed with the lowest cost.

10. The apparatus according to claim 9, characterized in that, The second determining unit determines the actual location information of the object based on the stored historical location information of the object and the real-time location information, including: For any object, if there is historical location information for the object and real-time location information for the object, the real-time location information of the object is updated based on the historical location information to determine the actual location information of the object. If historical location information for an object exists, but real-time location information for the object does not exist, the actual location information of the object shall be determined based on the historical location information of the object. If there is no historical location information for the object, but there is real-time location information for the object, the actual location information of the object is determined based on the real-time location information of the object. And / or, The third determining unit determines the sorting result of each object based on the stored historical sorting results of the objects and the actual location information, including: For any two objects, based on the historical sorting results of the two objects, add a first offset to the actual position information of the object that is sorted first, and add a second offset to the actual position information of the object that is sorted second. Based on the updated actual position information of the two objects, determine the sorting result of the two objects. The first offset is greater than the second offset, and the larger the offset, the greater the forward shift of the position. And / or, The third determining unit determines the sorting result of each object based on the stored historical sorting results of the objects and the actual location information, including: For any two objects, if the maximum area of ​​the overlapping region of the two objects exceeds a preset area threshold during the forward movement of the other object, the other object is determined to be sorted after the sorting of the first object. And / or, The third determining unit determines the sorting result of each object based on the stored historical sorting results of the objects and the actual location information, including: If the distance between the frontmost edges of multiple objects is less than a preset distance threshold, the sorting results of these multiple objects will be used as candidate sorting results for these multiple objects. And / or, The separation processing unit determines the optimal module speed based on the sorting results of the objects, including: When there is only one sorting result, the module speed with the lowest cost under that sorting result is determined as the optimal module speed. And / or, The separation processing unit determines the optimal module speed based on the sorting result of each object, and performs object separation processing based on the optimal module speed, including: When there are multiple sorting results, determine the target module speed with the lowest cost for each sorting result; Based on the cost of the target module speed, the target module speed with the lowest cost is determined as the optimal module speed; And / or, The separation processing unit performs object separation processing based on the optimal module speed, including: For any control module covered by any object, the speed of the control module is controlled to be consistent with the determined optimal module speed.

11. An electronic device, characterized in that, The method includes a processor and a memory, the memory storing machine-executable instructions that can be executed by the processor, the processor executing the machine-executable instructions to implement the method as described in any one of claims 1-8.

12. A machine-readable storage medium, characterized in that, The machine-readable storage medium stores machine-executable instructions, which, when executed by a processor, implement the method as described in any one of claims 1-8.

Images