Programmable remote control platform for modular robots

Abstract

The application provides a programmable remote control platform for a combined robot, and relates to the technical field of robot control.The method comprises the following steps: a task analysis module is used to obtain a target control task according to a host computer control script; a geometric positioning module is used to obtain a work image according to the target control task, and a target lateral offset and a target longitudinal distance are obtained from a target grounding point and a ground calibration mapping relationship; a parameter acquisition module is used to obtain a first parameter from the target lateral offset and a second parameter from the target longitudinal distance.The application can reduce the control level error adjustment caused by single-frame target position fluctuation, and improve the stability of the combined robot in executing the programmable remote control task.

Description

Technical Field

[0001] This invention relates to the field of robot control technology, and in particular to a programmable remote control platform for modular robots. Background Technology

[0002] Modular robots are typically composed of a mobile chassis, sensing components, and execution components. They are controlled by a host computer that issues control scripts to enable the robot to perform operations such as moving, approaching, and aligning with the target according to the task sequence specified in the script. When the robot is performing a target approach task, the remote control platform needs to determine the target position based on the work image acquired by the front-end image acquisition device and generate steering control commands and forward control commands accordingly.

[0003] Existing remote control platforms typically select the robot motion control level based on the center position or lateral pixel deviation of the target entity in a single frame of the operation image. However, the position of the target entity in the operation image is affected by the target distance, the installation angle of the camera device, the imaging perspective relationship, and fluctuations in the target boundary recognition. The image position corresponding to the same ground position may change. When the target entity is near adjacent control zones, the continuously acquired operation images are prone to position jumps, causing different steering correction levels or forward speed levels to be selected in adjacent control cycles. Summary of the Invention

[0004] This invention provides a programmable remote control platform for modular robots, which can reduce control level misadjustment caused by single-frame target position fluctuations and improve the stability of modular robots in performing programmable remote control tasks.

[0005] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows: A programmable remote control platform for a modular robot includes: The task parsing module is used to obtain the target control task based on the host computer control script; The geometric positioning module is used to acquire operation images according to the target control task, and obtain the target's lateral offset and longitudinal distance from the target's grounding point and ground calibration mapping relationship; The parameter acquisition module is used to obtain the first parameter from the target's lateral offset and the second parameter from the target's longitudinal distance. The interval division module is used to obtain the first parameter interval and the second parameter interval according to the preset parameter boundaries; The mapping query module is used to query the two-dimensional control mapping table according to the first parameter range and the second parameter range to obtain candidate control levels. The level adjustment module is used to adjust the candidate control level based on the continuous over-limit results of the first parameter in three consecutive frames to obtain the target control level; The execution control module is used to control the moving actuators according to the target control level to obtain the task execution status.

[0006] The above-described solution of the present invention has at least the following beneficial effects: The target grounding point is determined by the geometric positioning module, and the target's lateral offset and longitudinal distance are obtained by using the ground calibration mapping relationship. Candidate control levels can be determined based on the target entity's lateral and longitudinal positions. The candidate control levels are adjusted by comparing the first parameter with the first threshold in three consecutive frames, which can reduce the erroneous adjustment of control levels caused by single-frame target position fluctuations. The stability of the correspondence between the steering correction level and the forward speed level and the target position is improved, the repeated switching of control levels of the mobile actuator in the continuous control cycle is reduced, and the stability of the combined robot in performing programmable remote control tasks is improved. Attached Figure Description

[0007] Figure 1 This is a schematic diagram of a programmable remote control platform for a modular robot provided by an embodiment of the present invention.

[0008] Figure 2 This is a schematic diagram of the process of obtaining operation images and determining target grounding points according to target control tasks, and obtaining target lateral offset and target longitudinal distance according to ground calibration mapping relationship, provided by an embodiment of the present invention.

[0009] Figure 3 This is a schematic diagram of the process of controlling the moving actuator according to the target control level and obtaining the task execution status provided by an embodiment of the present invention. Detailed Implementation

[0010] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0011] like Figure 1 and Figure 3As shown, an embodiment of the present invention proposes a programmable remote control platform for a modular robot system composed of different mobile chassis, sensing components, and execution components. During the initialization phase, the platform reads the configuration description file of the modular robot. The configuration description file includes: mobile chassis type identifier, intrinsic and extrinsic parameters of the front image acquisition device, steering control range and forward control range of the mobile actuator, and ground calibration mapping relationship. For different combination configurations, the platform loads the corresponding configuration description file so that the two-dimensional control mapping table, steering control mapping table, and forward control mapping table are all adapted to the actual hardware characteristics of the current modular robot.

[0012] The task parsing module is used to obtain the target control task based on the host computer control script, including: The host computer control script refers to the script issued by the host computer to specify the execution order and task parameters of the modular robot. The host computer refers to the external control device that is connected to the modular robot through a communication link and issues control scripts to the modular robot.

[0013] The host computer control script is a pre-written task description file, including at least one task record. A task record refers to a section of content in the host computer control script used to describe an independent task. Each task record is located between a task start identifier and a task end identifier. The task start identifier is a preset character used to mark the beginning position of a task record, and the task end identifier is a preset character used to mark the end position of a task record.

[0014] Each task record includes a task identifier field, a target entity type field, a work area range field, a task priority field, and a work mode field. The fields refer to information items in the task record consisting of a field name and a field value. The task identifier field is used to distinguish different task records. The target entity type field is used to record the category name of the target entity to be identified. The work area range field is used to record the boundary of the area where the combined robot performs target approach or target alignment tasks. The task priority field is used to indicate the execution order between task records. The work mode field is used to distinguish between target approach mode and target alignment mode.

[0015] The task parsing module reads the task start identifier and task end identifier sequentially from the start position of the host computer control script. The content between the same task start identifier and task end identifier is identified as a task record. The module searches for each field name in the task record and identifies the characters after the field name, the next field name, or before the task end identifier as the corresponding field value. Leading and trailing spaces and newline characters are removed from the field values. In this way, the task identifier field value, target entity type field value, job area range field value, task priority field value, and job mode field value are obtained respectively.

[0016] The task parsing module compares the target entity type field value with the pre-stored target entity type correspondence table item by item. The target entity type correspondence table is a pre-stored lookup table used to convert the target entity category name into the target entity type identifier. Each item in the target entity type correspondence table includes the target entity category name and the target entity type identifier. When the target entity type field value is the same as the target entity category name in an item, the target entity type identifier in that item is determined as the target entity type identifier of the current task record. If there is no item with the same name, the current task record is determined as having an invalid target entity type.

[0017] The task parsing module splits the work area range field value into left horizontal boundary, right horizontal boundary, near vertical boundary, and far vertical boundary according to the internal arrangement order of the field. When the left horizontal boundary is smaller than the right horizontal boundary, the near vertical boundary is smaller than the far vertical boundary, and all boundaries are within the coverage of the ground calibration mapping relationship, the area enclosed by the four boundaries is determined as the target work area; otherwise, the current task record is determined as an invalid work area.

[0018] The task parsing module converts the task priority field value into a comparable task priority value, and compares the job mode field value with a pre-stored job mode mapping table to obtain the job mode identifier. The job mode mapping table is a pre-stored lookup table used to convert job mode names into job mode identifiers. If the job mode cannot be converted into a task priority value or the job mode identifier cannot be obtained, the current task record is determined to have invalid task attributes.

[0019] When the target entity type, work area, and task attributes are all valid, the task parsing module combines the task identifier, target entity type identifier, target work area, task priority value, and work mode identifier into a candidate control task. The candidate control task refers to the task record that has passed the validity verification and is waiting to be executed. When there are multiple candidate control tasks, they are arranged in descending order of task priority value. When the task priority values ​​are the same, they are arranged according to the order of the corresponding task records in the host computer control script. The candidate control task that is ranked first and has not yet been executed is determined as the target control task.

[0020] The geometric positioning module acquires operational images according to the target control task, and obtains the target's lateral offset and longitudinal distance from the target's grounding point and ground calibration mapping relationship, including: The task of controlling the target is to acquire images of the working area in front of the combined robot to obtain working images. The working images are images of the working area in front of the robot obtained by the front image acquisition device, which include the target entity and the ground area around the target entity. The front image acquisition device is a camera device installed at the front of the combined robot to acquire images in front of the robot.

[0021] The image acquisition device continuously acquires images of the working area in front of the robot according to a preset acquisition cycle, and writes the acquisition time for each frame of the working image. The acquisition cycle is determined according to the moving speed of the combined robot and the size of the target entity, so that the same target entity can be continuously identified in two adjacent frames of the working image. The target entity is identified based on the working image to obtain the target area. The target entity refers to the object that the combined robot needs to approach and align with, and the target area refers to the pixel area occupied by the target entity in the working image.

[0022] The specific process of identifying target entities is as follows: Perform target detection operation on the job image. Target detection operation refers to the operation of locating the location of the target entity in the job image, detecting whether the job image contains a target entity, and marking the pixel area covered by the target entity as the target area when the target entity is detected.

[0023] Target detection can be achieved using template matching, which involves pre-storing template images of the target entity. The template image is a reference image pre-acquired in a standard pose of the target entity. The matching degree between the current image region and the template image is calculated position by position in the working image. The matching degree can be measured by the gray-level similarity or texture feature similarity between the current image region and the template image. Gray-level similarity is determined by the consistency of gray-level values ​​at corresponding pixel positions in the two image regions, while texture feature similarity is determined by the consistency of the boundary direction distribution in the two image regions. The region with the highest matching degree is determined as the target region.

[0024] When multiple target regions are detected in the work image, delete the target regions that do not match the target entity type identifier in the target control task, then delete the target regions that exceed the target work area. If only one target region remains, determine that target region as the target region corresponding to the target entity. If multiple target regions remain, determine the target region with the smallest distance between the target center point and the target center point in the previous valid work image as the current target region. If there is no previous valid target center point, select the target region that is first in the order of the target regions in the work image from left to right and from top to bottom.

[0025] The previous valid operation image refers to the operation image that has undergone boundary validity verification and corner point validity verification and has determined the target ground point and target center point before the current operation image. The previous valid target center point refers to the target center point in the previous valid operation image. When no target area matching the target entity type identifier in the target control task is detected in the operation image, the target recognition failure result is output and the current operation image is not passed to the subsequent level adjustment process.

[0026] Boundary points are extracted along the boundary of the target region to obtain an ordered sequence of boundary points. Boundary points refer to pixels located on the contour of the target region, and the ordered sequence of boundary points refers to the sequence of boundary pixels arranged in the direction of the boundary of the target region.

[0027] The specific process for extracting boundary points is as follows: Starting from the topmost and leftmost boundary point of the target region, search for the next boundary point adjacent to the current boundary point in a clockwise direction until returning to the starting boundary point, forming an ordered boundary point sequence. When the boundary of the target region is not closed, there are mutually separated boundaries, or the number of boundary points is insufficient to form four position subsets, the current target region is determined to be an invalid boundary. For each boundary point in the ordered boundary point sequence, the first extension direction and the second extension direction are obtained according to the front boundary segment and the back boundary segment of the boundary point, respectively. The front boundary segment refers to a boundary segment that extends forward along the boundary direction with the boundary point as the endpoint, and the back boundary segment refers to a boundary segment that extends backward along the boundary direction with the boundary point as the starting point.

[0028] The front boundary segment consists of consecutive boundary points selected in the reverse direction of the ordered boundary point sequence, while the back boundary segment consists of the same number of consecutive boundary points selected in the forward direction of the ordered boundary point sequence. The number of boundary points contained in the front and back boundary segments is determined by the total length of the ordered boundary point sequence. The longer the boundary point sequence, the more boundary points the boundary segment contains, so that the boundary segment can reflect the local orientation of the boundary without being affected by the fluctuation of the position of a single boundary point.

[0029] The first extension direction is obtained by fitting a straight line to the front boundary segment. Specifically, based on the image coordinates of each boundary point in the front boundary segment, a fitting straight line is determined that minimizes the sum of the squares of the distances from each boundary point to the straight line. The orientation of the fitting straight line along the direction of the ordered boundary point sequence is the first extension direction. The second extension direction is obtained by fitting the rear boundary segment in the same way.

[0030] The change in direction is obtained by the angle between the first extension direction and the second extension direction. The change in direction refers to the angle between the first extension direction and the second extension direction, which is used to measure the degree of bending of the boundary of the target area at the boundary point. The larger the angle, the greater the degree of bending of the boundary at the boundary point.

[0031] The change in direction is taken as the smaller angle between the first extension direction and the second extension direction, so that the calculation of the change in direction is not affected by the direction extracted from the boundary point.

[0032] Corner threshold refers to the threshold for determining whether a boundary point is a corner. The corner threshold is determined using a calibration reference method. Specifically, the directional change of each boundary point in the calibration sample with pre-marked actual corner points is obtained. The median value between two adjacent directional changes is used as a candidate corner threshold. Corner detection is performed using each candidate corner threshold. The candidate corner threshold that can detect four corner points in each calibration sample and has the fewest incorrectly retained ordinary boundary points is selected as the corner threshold. The calibration sample refers to the target area sample image that has been pre-collected and manually marked with the positions of four actual corner points. Actual corner points refer to the actual corner points that exist on the contour of the target area.

[0033] Boundary points whose direction changes exceed the corner threshold are identified as candidate corner points, resulting in a candidate corner point set. Here, a candidate corner point is a boundary point whose direction changes exceed the corner threshold, and the candidate corner point set is the set of all candidate corner points.

[0034] Based on the positions of the candidate corner points in the upper left, upper right, lower right, and lower left of the target area, the set of candidate corner points is divided into four position subsets, which correspond to the upper left, upper right, lower right, and lower left regions of the target area, respectively.

[0035] The candidate corner point with the largest change in direction in each position subset is determined as the target corner point corresponding to that position subset, resulting in four target corner points: the upper left target corner point, the upper right target corner point, the lower right target corner point, and the lower left target corner point. When the change in direction is the same, the candidate corner point that is closer to the outer vertex of the corresponding position subset is selected. The outer vertex refers to the corner point position that is farthest from the center of the target area in the orientation of the target area corresponding to each position subset.

[0036] If no candidate corner point exists in any subset of locations, the current working image is determined as an invalid corner point, and the target grounding point is not determined based on the working image. The target center point is obtained by connecting the lines connecting the two sets of relative target corner points. That is, the first diagonal is obtained by connecting the upper left target corner point and the lower right target corner point, and the second diagonal is obtained by connecting the upper right target corner point and the lower left target corner point. The common image position of the two diagonals is determined as the target center point. The target center point refers to the pixel point at the center of the target area.

[0037] The lower edge of the target is obtained based on the lower left and lower right target corner points, and the target grounding point is obtained based on the midpoint of the lower edge of the target. The lower edge of the target refers to the lower boundary of the target area determined by the line connecting the lower left and lower right target corner points, and the target grounding point is the midpoint of the lower edge of the target, which is used to represent the actual position of the target entity on the ground. The horizontal image coordinates of the target grounding point are the average of the horizontal image coordinates of the lower left and lower right target corner points, and the vertical image coordinates of the target grounding point are the average of the vertical image coordinates of the lower left and lower right target corner points.

[0038] For target areas where the lower left and lower right target corners cannot be formed, the target area is determined as not applicable to this geometric positioning process, and the target ground point is not determined based on the target area. The target ground point is located at the intersection of the target entity and the ground. The correspondence between its image position and the actual position on the ground is more stable than that of the target center point. This is because the position of the target center point in the working image will shift significantly with the change of imaging perspective, while the target ground point is close to the ground and is less affected by the imaging perspective.

[0039] The ground calibration mapping relationship refers to the correspondence between the pre-calibrated pixel positions of the operation image and the actual positions on the ground. It is used to convert the pixel positions in the operation image into the actual positions in the ground coordinate system. The ground physical coordinate system takes the reference position at the front of the combined robot as the origin, the projection of the longitudinal center line of the combined robot on the ground as the vertical axis, and the direction perpendicular to the longitudinal center line and pointing to the right of the combined robot as the positive direction of the horizontal axis. The ground calibration mapping relationship is pre-established through the calibration reference method. Specifically, multiple calibration points with known positions are arranged in the ground coordinate system. Each calibration point is located on the same operation ground and covers the lateral and longitudinal ranges used by the combined robot when performing the target approach task. The front image acquisition device acquires a calibration image containing all calibration points, records the pixel coordinates of each calibration point in the calibration image and the corresponding ground physical coordinates of each calibration point, and solves the mapping parameters for converting the pixel coordinates of the operation image into ground coordinates based on the correspondence between the pixel coordinates of all calibration points and the ground physical coordinates, thus obtaining the ground calibration mapping relationship.

[0040] The ground calibration mapping relationship adopts homography transformation. Homography transformation refers to the perspective transformation that maps points on the plane of the work image to the plane of the ground. The ground coordinates of any pixel in the work image are obtained by homography transformation of the image coordinates of that pixel. According to the image position of the target ground point and the ground calibration mapping relationship, the image position of the target ground point is converted into ground coordinates, and the target lateral offset and target longitudinal distance are obtained from the ground coordinates. The target lateral offset is the lateral distance of the target ground point relative to the longitudinal centerline of the combined robot, and the target longitudinal distance is the longitudinal distance of the target ground point relative to the front reference position of the combined robot. The longitudinal centerline is the projection of the longitudinal axis of symmetry of the combined robot onto the ground, and the front reference position is the reference position of the front end of the combined robot on the ground. When the target ground point is outside the coverage area of ​​the ground calibration mapping relationship, the ground mapping out-of-bounds result is output, and the position outside the coverage area is not extrapolated.

[0041] The parameter acquisition module is used to obtain a first parameter from the target's lateral offset and a second parameter from the target's longitudinal distance, including: The lateral offset direction is obtained based on the position of the target's lateral offset relative to the longitudinal centerline of the combined robot. The lateral offset direction is used to indicate whether the target's grounding point is located to the left or right of the longitudinal centerline. When the target's grounding point is located on the longitudinal centerline, the lateral offset direction is centered.

[0042] A negative direction identifier is obtained when the lateral offset direction is to the left, and a positive direction identifier is obtained when the lateral offset direction is to the right. The direction identifier refers to the symbol used to indicate the left and right directions of the target grounding point relative to the longitudinal centerline. The first parameter is obtained based on the direction identifier and the distance value of the target lateral offset. The distance value refers to the absolute value of the target lateral offset, that is, the lateral distance of the target grounding point from the longitudinal centerline of the combined robot. The first parameter represents the directional lateral offset of the target grounding point relative to the longitudinal centerline. The negative direction identifier corresponds to a negative value for the first parameter, and the positive direction identifier corresponds to a positive value for the first parameter. The first parameter is 0 when the target grounding point is located on the longitudinal centerline. The second parameter is obtained based on the target longitudinal distance. The second parameter represents the longitudinal distance of the target grounding point relative to the front reference position of the combined robot.

[0043] The value ranges of the first and second parameters are obtained based on the horizontal and vertical ranges covered by the ground calibration mapping relationship. The horizontal range refers to the horizontal interval of the ground that can be accurately mapped by the ground calibration mapping relationship, and the vertical range refers to the vertical interval of the ground that can be accurately mapped by the ground calibration mapping relationship. The value range of the first parameter is determined by the horizontal range, and the value range of the second parameter is determined by the vertical range. When either parameter exceeds its corresponding value range, the output parameter out-of-bounds result is not executed and the interval query is not performed.

[0044] The interval division module is used to obtain a first parameter interval and a second parameter interval according to preset parameter boundaries, including: Based on the range of values ​​of the first parameter and multiple first boundary values, multiple consecutive intervals of the first parameter are obtained. The first boundary value refers to the dividing value that divides the range of values ​​of the first parameter into multiple consecutive intervals, and the first parameter interval refers to the consecutive range of values ​​of the first parameter defined by adjacent first boundary values.

[0045] The first boundary value is determined using a calibration reference method. The calibration process is as follows: the platform arranges the executable steering control commands of the mobile actuator in ascending order of actual steering response. The steering control command refers to the command used to control the steering direction and steering amplitude of the mobile actuator. The actual steering response refers to the actual steering angle generated by the mobile actuator after executing the steering control command. For two adjacent steering control commands, multiple target entities with different lateral positions are set in the ground calibration mapping area. Each entity executes one control cycle and obtains the first parameter after control. The maximum lateral position that can reduce the absolute value of the first parameter without reversing the sign of the first parameter is the lower-level endpoint. The minimum lateral position that can further reduce the absolute value of the first parameter without reversing the sign of the first parameter is the higher-level endpoint. The midpoint between the two endpoints is determined as the candidate value of the first boundary. Reversing the sign means that the sign of the first parameter changes from positive to negative or from negative to positive.

[0046] First boundary candidate values ​​are obtained at different longitudinal positions, and the candidate values ​​that ensure no sign reversal at all calibration positions and have smaller absolute values ​​of the first parameter after control are selected as the first boundary values. The above process is performed on the left and right sides respectively. The first boundary value on the left is negative and the first boundary value on the right is positive. All first boundary values ​​are arranged in ascending order to obtain multiple consecutive first parameter intervals.

[0047] The second boundary value is determined using a calibration reference method. The calibration process involves the platform sending forward control commands corresponding to different forward speed levels to the mobile actuator, recording the actual forward distance within one control cycle and the braking distance after stopping the drive. The forward control command refers to the command used to control the forward speed of the mobile actuator, and the braking distance refers to the distance that the combined robot continues to move forward due to mechanical inertia after receiving the stop command. The actual forward distance and the braking distance are added together to obtain the remaining longitudinal distance required for the corresponding forward speed level. The midpoint between the remaining longitudinal distances required for two adjacent forward speed levels is determined as the candidate value of the second boundary. The calibration samples located on both sides of the candidate value of the second boundary are used for verification. If a higher speed level causes the combined robot to cross the target position, the candidate value of the second boundary is moved away from the target entity until the calibration samples on both sides of the candidate value have not crossed the target position, thus obtaining the second boundary value.

[0048] Based on the range of values ​​of the second parameter and multiple second boundary values, multiple consecutive intervals of the second parameter are obtained. The second boundary value refers to the dividing value that divides the range of values ​​of the second parameter into multiple consecutive intervals, and the second parameter interval refers to the consecutive range of values ​​of the second parameter defined by adjacent second boundary values.

[0049] Each steering correction level corresponds to a preset steering angle value. The higher the level, the larger the corresponding steering angle value. The steering angle value corresponding to each level is determined by the proportional reference method, that is, the maximum steering angle allowed by the moving actuator is used as the reference value, and the maximum steering angle is divided into the steering angle values ​​corresponding to each level according to the number of steering correction levels.

[0050] Each gear in the forward speed level corresponds to a preset forward speed value. The higher the gear, the greater the forward speed value. The forward speed value corresponding to each gear is determined by the proportional benchmark method, that is, the maximum forward speed allowed by the moving actuator is used as the benchmark, and the maximum forward speed is proportionally divided into the forward speed values ​​corresponding to each gear according to the number of gears in the forward speed level.

[0051] The current first parameter interval is obtained based on the comparison results between the first parameter and each first boundary value, and the current second parameter interval is obtained based on the comparison results between the second parameter and each second boundary value. The current first parameter interval refers to the first parameter interval into which the current value of the first parameter falls, and the current second parameter interval refers to the second parameter interval into which the current value of the second parameter falls.

[0052] When any parameter is located at a common boundary, the parameter is assigned to the parameter interval with the common boundary as its lower boundary. The common boundary refers to the boundary value shared by two adjacent parameter intervals. That is, the parameter interval with the common boundary as its lower boundary is the interval with the larger value among the two adjacent parameter intervals. When the parameter is equal to the upper limit of the value range, the parameter is assigned to the last parameter interval.

[0053] The mapping query module is used to query the two-dimensional control mapping table according to the first parameter range and the second parameter range to obtain candidate control levels, including... A two-dimensional control mapping table is a pre-constructed two-dimensional table with a first parameter range as the row index and a second parameter range as the column index, used to query the corresponding control level based on the first parameter range and the second parameter range.

[0054] The steering correction level is determined by the actual steering response of the mobile actuator. Specifically, the platform sends multiple candidate steering control quantities to the mobile actuator. The candidate steering control quantity refers to the optional output quantity used to control the steering of the mobile actuator. The actual steering response generated by each candidate steering control quantity is obtained based on motion feedback. Candidate steering control quantities that cannot generate a identifiable steering response are not written into the level sequence. When the difference between the actual steering responses generated by two candidate steering control quantities cannot be distinguished by motion feedback, the two candidate steering control quantities are merged into the same steering correction level, and the candidate steering control quantity with the smaller output amplitude is retained. The remaining candidate steering control quantities are arranged in ascending order of actual steering response to obtain the steering correction level sequence. Each level is not required to use the same steering correction amplitude difference.

[0055] The forward speed level is determined by the actual speed response of the mobile actuator. Specifically, the platform sends multiple candidate forward control quantities, which are optional output quantities used to control the forward speed of the mobile actuator. The actual forward speed is obtained based on motion feedback. Candidate forward control quantities that cannot generate effective forward motion are not written into the level sequence. Candidate forward control quantities whose actual forward speed cannot be distinguished by motion feedback are merged into the same forward speed level, and candidate forward control quantities with smaller output amplitudes are retained. The remaining candidate forward control quantities are arranged in ascending order of actual forward speed to obtain the forward speed level sequence. Each forward speed level is not required to use the same speed difference. Multiple first parameter intervals are set as row indices according to their numerical order, that is, in the order of the first parameter from smallest to largest value, each first parameter interval is arranged as a row of a two-dimensional control mapping table. The row index is used to identify a row in the two-dimensional control mapping table.

[0056] Multiple second parameter intervals are set as column indices according to their numerical order, that is, the second parameter intervals are arranged sequentially as columns of the two-dimensional control mapping table according to the order of the second parameter from smallest to largest value. The column index is used to identify a column in the two-dimensional control mapping table. Each row index is associated with the corresponding steering direction and steering correction level, where the steering direction refers to the direction of the combined robot's steering, including left turn and right turn. The first parameter interval with a negative first parameter value is associated with the left turn, and the first parameter interval with a positive first parameter value is associated with the right turn. Each column index is associated with the corresponding forward speed level.

[0057] In the two-dimensional control mapping table, each row and column intersection position pre-stores the control level, which consists of steering direction, steering correction level, and forward speed level. The control level refers to the combination of steering and forward speed used by the combined robot in one control cycle. The specific structure of the two-dimensional control mapping table is shown in Table 1 below.

[0058] Table 1 shows the structure of the two-dimensional control mapping table:

[0059] In Table 1, the left rows correspond to the first parameter intervals where the first parameter is negative. The larger the lateral offset, the higher the steering correction level. The right rows correspond to the first parameter intervals where the first parameter is positive. The larger the lateral offset, the higher the steering correction level. Each column corresponds to a different forward speed level. The larger the longitudinal distance, the higher the forward speed level. The control level in each cell is composed of the steering direction and steering correction level of the row in which the cell is located, as well as the forward speed level of the column in which the cell is located.

[0060] The cell content in the two-dimensional control mapping table is determined as follows: the larger the horizontal offset corresponding to the first parameter, the higher the steering correction level associated with the row index, and the horizontal offset direction determines the steering direction associated with the row index. The larger the vertical distance corresponding to the second parameter, the higher the forward speed level associated with the column index. Based on this, the steering direction and steering correction level corresponding to each row index are combined with the forward speed level corresponding to each column index and stored in the corresponding intersection position.

[0061] For each row and column intersection, a candidate control level is executed using calibration samples located within the corresponding first parameter interval and second parameter interval. When the sign of the first parameter is reversed after executing the candidate control level and the first parameter does not fall into the first parameter interval including zero, the candidate control level is determined as over-steering. When the second parameter crosses the termination position specified by the target control task after executing the candidate control level, the candidate control level is determined as over-forwarding. When the target entity moves out of the operation image or out of the coverage area of ​​the ground calibration mapping relationship, the candidate control level is determined as positioning failure.

[0062] Candidate control levels that exhibit oversteering, over-forward, or positioning failure are deleted. For the remaining candidate control levels, the absolute values ​​of the first parameter after control are obtained for all calibration samples, and the largest absolute value of the first parameter is determined as the maximum residual lateral offset. Candidate control levels with smaller maximum residual lateral offsets are selected first. If the maximum residual lateral offsets are the same, the candidate control level with a larger reduction in the second parameter after control is selected. If the reduction in the second parameter is still the same, the candidate control level with a lower forward speed level is selected. The selected steering direction, steering correction level, and forward speed level are written to the corresponding row and column intersection positions. If no candidate control level meets the conditions, the corresponding row and column intersection position is marked as unavailable, and the control level of the adjacent row and column intersection position is not used as a substitute.

[0063] The current row index is obtained based on the current first parameter range, the current column index is obtained based on the current second parameter range, and the control level stored at the corresponding intersection position in the two-dimensional control mapping table is queried based on the current row index and the current column index to obtain the candidate control level. The candidate control level refers to the control level initially determined before continuous frame adjustment. If the corresponding row and column intersection position is empty or marked as unavailable, the mapping query result will be output as a failure.

[0064] The level adjustment module is used to adjust the candidate control level based on the continuous exceedance results of the first parameter in three consecutive frames to obtain the target control level, including: Three consecutive operation images are obtained based on the current operation image and the two operation images preceding it. The three consecutive operation images refer to the operation image of the current control cycle and the operation images of the two adjacent control cycles preceding it.

[0065] Arrange the three work images in order of acquisition time from earliest to latest, and compare the acquisition time interval between two adjacent frames. If the acquisition time interval is not greater than the allowed acquisition interval in the current image acquisition configuration, a continuous acquisition result is obtained. If any acquisition time interval exceeds the allowed acquisition interval, an acquisition interruption result is obtained. The corresponding three work images are not used for continuous over-limit judgment. The allowed acquisition interval refers to the maximum allowable difference in acquisition time between two adjacent work images.

[0066] Three first parameters are obtained based on the same target entity in three consecutive operation images. That is, for each of the three consecutive operation images, the first parameters corresponding to that frame are obtained according to the processing method of the aforementioned geometric positioning module and parameter acquisition module. The same target entity is associated through recognition type, target region overlap degree and target feature similarity degree. Here, recognition type refers to the target entity category output by the target detection operation. Only target regions with the same recognition type in adjacent operation images are formed into candidate target pairs. Candidate target pairs refer to two target regions in two adjacent operation images that may belong to the same target entity.

[0067] The degree of overlap of target regions is determined by the intersection area and the union area of ​​two target regions. Specifically, the intersection area is obtained by obtaining the number of pixels that are covered by both target regions, and the union area is obtained by obtaining the number of pixels covered by the merged target regions. The degree of overlap of target regions is obtained by comparing the intersection area and the union area. When there are no pixels that are covered by each other, the degree of overlap of target regions is determined to be the minimum result.

[0068] The similarity of target features is determined based on the distribution of the boundary extension directions of corresponding image positions in the two target regions. Specifically, the two target regions are adjusted to the same image size and divided into multiple sub-regions in the same way. The number of boundary points in different boundary extension directions in each sub-region is counted to obtain two directional distribution sequences. The data at corresponding positions in the two directional distribution sequences are compared in turn. The smaller the difference, the higher the similarity of target features.

[0069] The region overlap threshold and target feature similarity threshold are determined by continuous image samples marked with the same target entity correspondence. Specifically, the degree of overlap of target regions and the degree of target feature similarity of real target pairs and non-target pairs are calculated respectively. The midpoint between two adjacent different results is used as a candidate threshold. The candidate threshold combination that can retain all real target pairs is selected first, and then the candidate threshold combination with fewer incorrectly associated non-target pairs is selected. When the number of incorrect associations is the same, the candidate threshold combination with higher region overlap threshold and higher target feature similarity threshold is selected in turn.

[0070] During runtime, when the overlap of target regions reaches the region overlap threshold and the similarity of target features reaches the target feature similarity threshold, the corresponding candidate target pairs are determined to be the same target entity. When there are multiple candidate target pairs that meet the conditions, the candidate target pairs with larger overlap of target regions, larger similarity of target features, and smaller distance between target center points are selected in sequence. When there are no candidate target pairs that meet the conditions, three consecutive frames are determined to be invalid target associations.

[0071] The direction consistency result is obtained based on the positive and negative directions of the three first parameters, where the direction consistency result is used to indicate whether the direction identifiers of the three first parameters are consistent.

[0072] The distance exceedance result is obtained by comparing the absolute values ​​of the three first parameters with the first threshold. The distance exceedance result is used to indicate whether the absolute values ​​of the three first parameters all exceed the first threshold.

[0073] The first threshold is the lateral offset distance threshold used to determine whether the absolute value of the first parameter exceeds the limit. The first threshold is determined by the proportional reference method, taking the lateral range covered by the ground calibration mapping relationship as the reference quantity, and taking a preset proportion of the reference quantity as the first threshold. The preset proportion is determined according to the steering response accuracy of the mobile actuator, so that when the distance of the target ground point deviating from the longitudinal centerline exceeds the first threshold, the combined robot needs to increase the steering correction level.

[0074] The continuous over-limit result is obtained based on the direction consistency result and the distance over-limit result. The continuous over-limit result is used to indicate whether the first parameter of three consecutive frames is in the same direction and the distance exceeds the limit.

[0075] When the continuous over-limit result is over-limit and the steering correction level in the candidate control level is lower than the highest level, the steering direction and forward speed levels in the candidate control level remain unchanged, and the steering correction level is increased by one level to obtain the target control level. When the continuous over-limit result is not over-limit or the steering correction level in the candidate control level has reached the highest level, the candidate control level is determined as the target control level. The target control level refers to the control level ultimately used to control the moving actuator, and the highest level refers to the highest gear in the steering correction level gear. The specific method for determining the continuous over-limit result is as follows: an ordered sequence of first parameters is obtained based on the acquisition time of the three first parameters, where the ordered sequence of first parameters refers to the three first parameters arranged in chronological order of acquisition time; three direction identifiers are obtained based on the positive and negative directions of each first parameter in the ordered sequence, that is, each first parameter in the ordered sequence is judged to be a negative or positive direction identifier in turn; and three absolute values ​​are obtained based on the absolute values ​​of each first parameter in the ordered sequence.

[0076] When all three absolute values ​​are greater than the first threshold, the distance exceeding the limit is obtained for all of them. When at least one of the three absolute values ​​is not greater than the first threshold, the distance exceeding the limit for some of them is obtained.

[0077] A consistent direction consistency result is obtained when all three direction identifiers are negative or all three are positive. An inconsistent direction consistency result is obtained when all three direction identifiers contain both negative and positive identifiers or when any of the first parameters is 0. When all three first parameters are 0, the absolute values ​​of the three first parameters are not greater than the first threshold, resulting in a distance overshoot result that does not exceed the limit completely, and the steering correction level is not triggered. When the distance overshoot result is all overshoot and the direction consistency result is consistent, a continuous overshoot result that exceeds the limit is obtained; otherwise, a continuous overshoot result that does not exceed the limit is obtained. When the continuous overshoot result is not overshoot, the acquisition time is discontinuous, the target association is invalid, or the continuous frame validity result is invalid, the candidate control level is determined as the target control level. When the continuous overshoot result is overshoot but the steering correction level has reached the highest level, the candidate control level is maintained, and the state of the steering correction level reaching the upper limit is recorded in the task execution status.

[0078] The execution control module is used to control the moving actuator according to the target control level and obtain the task execution status, including: The steering control command is obtained based on the steering direction and steering correction level in the target control level. The steering control command is a command used to control the steering direction and steering amplitude of the moving actuator. Specifically, the steering control correspondence table is queried with the steering correction level as an index to obtain the corresponding steering control quantity. The steering control correspondence table is a pre-constructed lookup table used to convert the steering correction level into the steering control quantity. The steering direction and the selected steering control quantity are combined into the steering control command.

[0079] The forward control command is obtained based on the forward speed level in the target control level. The forward control command is a command used to control the forward speed of the moving actuator. Specifically, the forward control correspondence table is queried with the forward speed level as an index to obtain the corresponding forward control quantity. The forward control correspondence table is a pre-constructed lookup table used to convert the forward speed level into the forward control quantity. The forward control command is generated based on the selected forward control quantity.

[0080] The robot motion control command is obtained from the steering control command and the forward control command. The robot motion control command is a combination of commands used to control the movement of the mobile actuator in the two degrees of freedom of steering and forward movement.

[0081] The robot motion control command is sent to the mobile actuator, and motion feedback is received from the mobile actuator. Motion feedback refers to the feedback data returned by the mobile actuator after executing the robot motion control command, which reflects its actual motion. Motion feedback includes command reception status, turning feedback, forward feedback, and execution completion status. Command reception status refers to the feedback information of whether the mobile actuator has correctly received the robot motion control command. Turning feedback refers to the actual turning angle returned by the mobile actuator. Forward feedback refers to the actual forward speed returned by the mobile actuator. Execution completion status refers to the feedback information of whether the mobile actuator has completed all the motions indicated by the current robot motion control command.

[0082] The actual movement state of the combined robot is obtained based on motion feedback. When the mobile actuator can return to the position feedback, the start position and end position of the control cycle are obtained. The actual movement distance and actual movement direction are obtained based on the distance and direction between the two positions. When the mobile actuator returns to the speed feedback, the speed feedback within the control cycle is arranged according to the acquisition time. Each speed feedback is combined with the corresponding duration to form a segmented movement distance. The segmented movement distances are added together to obtain the actual movement distance. The actual movement direction is obtained based on the turning feedback. If motion feedback is missing, the acquisition time is reversed, or the motion feedback cannot correspond to the robot's motion control command, the actual movement state is determined to be invalid. The actual movement state refers to the actual position and actual posture of the combined robot after executing the robot's motion control command.

[0083] The task execution status is generated based on the actual movement status and sent to the host computer. The task execution status refers to the status information used to reflect the combined robot's execution of the target control task. The task execution status includes the task identifier of the target control task, the target control level, the actual movement status, and the instruction execution result. When the actual movement status is valid and the mobile actuator returns to the execution completion status, the instruction execution result is determined to be execution completion. When the mobile actuator rejects the robot's motion control instruction, the motion feedback does not return after the current control cycle, or the actual movement status is invalid, the instruction execution result is determined to be execution abnormal.

[0084] The preset motion correspondence is established through motion calibration before operation. The preset motion correspondence refers to the correspondence between the motion of the combined robot itself and the positional changes of the target entity in the operation image. Specifically, stationary target entities are set at different lateral and longitudinal positions within the coverage area of ​​the ground calibration mapping relationship. The combined robot is controlled to execute each available steering correction level and forward speed level. The actual movement state before and after control, as well as the changes in the lateral and longitudinal image positions of the target center point, are recorded. Records with the same steering direction, steering correction level, and forward speed level are grouped into the same motion state group. The motion state group refers to the set of calibration records with the same steering direction, steering correction level, and forward speed level. Records in which the target entity moves, the target recognition type changes, the target area is incomplete, or the image acquisition configuration is inconsistent are deleted.

[0085] For each motion state group, the minimum and maximum values ​​of the lateral position change of the target center point are obtained to form the allowable lateral change range, and the minimum and maximum values ​​of the longitudinal position change are obtained to form the allowable longitudinal change range. The allowable change direction is obtained based on the positive and negative combinations of the lateral and longitudinal position changes. Both ends of the range belong to the allowable change range. If there is no valid calibration record for the corresponding motion state group, the corresponding preset motion relationship is not established.

[0086] The validity of consecutive frames in the next control cycle is determined based on the task execution status. Specifically, a consecutive frame sequence is obtained based on the current operation image of the next control cycle and the two operation images preceding it. The consecutive frame sequence refers to the sequence consisting of the current operation image of the next control cycle and the operation images of the two adjacent control cycles preceding it. The recognition type of the target entities in the three frames of the consecutive frame sequence is compared to obtain a type consistency result. The recognition type refers to the target entity category output by the target detection operation. A consistent type consistency result is obtained when the three recognition types are the same and correspond to the target entity type identifier in the target control task; otherwise, an inconsistent type consistency result is obtained.

[0087] The target position change result is obtained based on the position of the target center point in the continuous frame sequence. Specifically, the horizontal and vertical position changes of the target center point in the second frame relative to the target center point in the first frame are determined as the first position change, and the horizontal and vertical position changes of the target center point in the third frame relative to the target center point in the second frame are determined as the second position change. If no valid target center point is formed in any frame, the target position change result is determined as invalid.

[0088] Based on the actual movement status in the task execution state, the preset motion correspondence is queried to obtain the allowed change direction, allowed lateral change range, and allowed longitudinal change range. When both the first position change and the second position change conform to the allowed change direction, and both the lateral position change and the longitudinal position change are within the corresponding allowed range, a consistent motion consistency result is obtained. When the preset motion correspondence does not exist, the version is inconsistent, the target position change result is invalid, or any position change exceeds the corresponding allowed range, an inconsistent motion consistency result is obtained. Among them, version inconsistency means that the currently used preset motion correspondence does not match the ground calibration mapping relationship or control strategy version number adopted in the current control cycle.

[0089] When both the type consistency result and the motion consistency result are consistent, the consecutive frame validity result is determined to be valid; otherwise, it is determined to be invalid. When the consecutive frame validity result is valid, the consecutive frame sequence is sent to the level adjustment module; when the consecutive frame validity result is invalid, the corresponding consecutive frames are not used to increase the steering correction level, and the operation images that cause type inconsistency or motion inconsistency are deleted.

[0090] When re-acquiring work images, the continuous frame sequence is updated in a rolling manner according to the acquisition time. The newly acquired work image is added to the continuous frame sequence as the latest frame. After more than three frames, the work image with the earliest acquisition time is deleted. After each update, the acquisition interval, target entity recognition type and target center point change of adjacent work images are re-checked. Historical work images in the continuous frame sequence that have exceeded the allowed acquisition interval are not retained for a long time.

[0091] The acquisition continues until the effective period of the current robot motion control command ends. The effective period is jointly defined by the current control cycle and the allowable no-feedback duration of the mobile actuator, and the shorter of the two periods is used. If a valid continuous frame sequence cannot be formed after the effective period ends, a continuous frame acquisition failure status is generated, the steering correction level is no longer increased, a stop motion command is sent to the mobile actuator, and the continuous frame acquisition failure status is sent to the host computer.

[0092] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A programmable remote control platform for modular robots, characterized in that, include: The task parsing module is used to obtain the target control task based on the host computer control script; The geometric positioning module is used to acquire operation images according to the target control task, and obtain the target's lateral offset and longitudinal distance from the target's grounding point and ground calibration mapping relationship; The parameter acquisition module is used to obtain the first parameter from the target's lateral offset and the second parameter from the target's longitudinal distance. The interval division module is used to obtain the first parameter interval and the second parameter interval according to the preset parameter boundaries; The mapping query module is used to query the two-dimensional control mapping table according to the first parameter range and the second parameter range to obtain candidate control levels. The level adjustment module is used to adjust the candidate control level based on the continuous over-limit results of the first parameter in three consecutive frames to obtain the target control level; The execution control module is used to control the moving actuators according to the target control level to obtain the task execution status.

2. The programmable remote control platform for modular robots according to claim 1, characterized in that, Based on the target control task, acquire operational images and determine the target grounding point. Obtain the target's lateral offset and longitudinal distance based on the ground calibration mapping relationship, including: Based on the target control task, the working area in front of the combined robot is acquired to obtain the working image; Identify target entities from the work image to obtain the target region; Based on the change in the boundary direction of the target area, four target corner points are obtained; Based on the positions of the four target corner points, the lower left and lower right target corner points are obtained; The lower edge of the target is obtained based on the lower left and lower right corner points; The target's grounding point is obtained based on the midpoint of the target's lower edge; Based on the image location of the target grounding point and the ground calibration mapping relationship, the target's lateral offset and longitudinal distance are obtained.

3. The programmable remote control platform for modular robots according to claim 2, characterized in that, Based on the changes in the boundary direction of the target area, four target corner points and the target center point are obtained, including: Boundary points are extracted along the boundary of the target region to obtain an ordered sequence of boundary points; Based on the front and back boundary segments of each boundary point in the ordered boundary point sequence, the first extension direction and the second extension direction are obtained. The change in direction is obtained based on the angle between the first extension direction and the second extension direction; A set of candidate corner points is obtained based on the boundary points whose direction changes exceed the corner point threshold; Based on the positions of the candidate corner points in the upper left, upper right, lower right, and lower left parts of the target area, four position subsets are obtained; Based on the candidate corner points with the largest directional changes in each position subset, four target corner points are obtained; The center point of the target is obtained by the intersection of the lines connecting the two sets of relative target corner points.

4. The programmable remote control platform for modular robots according to claim 3, characterized in that, The first and second parameters are obtained based on the target's lateral offset and longitudinal distance, including: The lateral offset direction is obtained based on the position of the target's lateral offset relative to the longitudinal centerline of the combined robot; When the lateral offset direction is to the left, a negative direction indicator is obtained; when the lateral offset direction is to the right, a positive direction indicator is obtained. The first parameter is obtained based on the direction indicator and the distance value of the target's lateral offset; the second parameter is obtained based on the target's longitudinal distance; wherein, the first parameter represents the lateral offset of the target's grounding point relative to the longitudinal centerline, and the second parameter represents the longitudinal distance of the target's grounding point relative to the robot's front reference position; Based on the horizontal and vertical ranges covered by the ground calibration mapping relationship, the value ranges of the first and second parameters are obtained.

5. The programmable remote control platform for modular robots according to claim 4, characterized in that, The first parameter interval and the second parameter interval are obtained based on the preset parameter boundaries, including: Based on the range of values ​​for the first parameter and multiple first boundary values, multiple consecutive intervals of the first parameter are obtained; Based on the range of values ​​for the second parameter and multiple second boundary values, multiple consecutive intervals of the second parameter are obtained; Based on the comparison results between the first parameter and each first boundary value, the current interval of the first parameter is obtained; Based on the comparison results between the second parameter and each second boundary value, the current interval of the second parameter is obtained; When any parameter is located at a common boundary, the parameter is assigned to the parameter interval with that common boundary as its lower boundary.

6. The programmable remote control platform for a modular robot according to claim 5, characterized in that, By querying the two-dimensional control mapping table based on the first and second parameter intervals, candidate control levels are obtained, including: Set multiple first parameter ranges as row indices in numerical order; Set multiple second parameter ranges as column indices in numerical order; Each row index is associated with the corresponding turning direction and turning correction level; Each column index is associated with a corresponding forward speed level; Each row and column intersection position pre-stores a control level consisting of steering direction, steering correction level, and forward speed level; The current row index is obtained based on the current first parameter range, and the current column index is obtained based on the current second parameter range; The candidate control levels are obtained by querying the corresponding control levels based on the current row index and the current column index.

7. The programmable remote control platform for modular robots according to claim 6, characterized in that, The target control level is obtained based on the candidate control level and the first parameter of three consecutive frames, including: Based on the current task image and the two task images preceding it, obtain three consecutive task images; Three first parameters are obtained based on the same target entity in three consecutive frames of the operation image; Based on the positive and negative directions of the three first parameters, the directional consistency result is obtained; The distance exceeding the limit result is obtained by comparing the absolute values ​​of the three first parameters with the first threshold. Based on the directional consistency results and the distance exceeding the limit results, the continuous exceeding the limit results are obtained; When the continuous exceedance result is an exceedance and the steering correction level in the candidate control level is lower than the highest level, the steering correction level is increased by one level to obtain the target control level. Otherwise, the candidate control level will be determined as the target control level.

8. The programmable remote control platform for a modular robot according to claim 7, characterized in that, Determining consecutive exceedance results includes: Based on the acquisition time of the three first parameters, an ordered sequence of first parameters is obtained; Based on the positive and negative directions of each first parameter in the ordered first parameter sequence, three direction identifiers are obtained; Based on the absolute values ​​of each first parameter in the ordered first parameter sequence, three absolute values ​​are obtained; When all three absolute values ​​are greater than the first threshold, the distance exceeding the limit for all items is obtained; otherwise, the distance exceeding the limit for some items is obtained. When all three direction indicators are negative or all three are positive, a consistent direction result is obtained; otherwise, an inconsistent direction result is obtained. When all distance exceedances are exceeded and the direction consistency is consistent, a continuous exceedance result is obtained; otherwise, a continuous exceedance result without exceedance is obtained.

9. The programmable remote control platform for a modular robot according to claim 8, characterized in that, Control the moving actuators according to the target control level and obtain the task execution status, including: Based on the steering direction and steering correction level in the target control level, the steering control command is obtained; Based on the forward speed level in the target control level, the forward control command is obtained; Based on the steering control command and the forward control command, the robot motion control command is obtained; Send robot motion control commands to the mobile actuator and receive motion feedback from the mobile actuator; Based on motion feedback, the actual movement state of the combined robot is obtained; The task execution status is generated based on the actual movement status and then sent to the host computer.

10. The programmable remote control platform for a modular robot according to claim 9, characterized in that, Determining the validity of consecutive frames in the next control cycle based on the task execution status includes: Based on the current operation image of the next control cycle and the two operation images preceding it, a continuous frame sequence is obtained; By comparing the identification types of target entities in three consecutive frames of a frame sequence, a type consistency result is obtained; Based on the position of the target's center point in a continuous frame sequence, the result of the target's position change is obtained; Based on the actual movement state in the task execution state and the preset movement correspondence, the allowable change direction and allowable change range of the target center point are obtained; Based on the comparison between the target position change results and the allowable change direction and allowable change range, the motion consistency result is obtained; Based on the type consistency results and motion consistency results, the validity results of consecutive frames are obtained.

Images