Information control methods, devices, electronic equipment and storage media

Abstract

This application discloses an information control method, apparatus, electronic device, and storage medium. The method includes: acquiring the working pose and relative positional relationship of a working device; determining a target point in response to a selection operation applied to a scene image; controlling the working device to perform a picking operation at a first position; calculating a second position based on target point cloud data and a preset point cloud model of the designated space; determining a delivery position based on the second position and the relative positional relationship, and controlling the working device to perform a delivery operation based on the delivery position to deliver the picked-up object to the designated space. The entire process only requires the user to select a target point in the scene image to achieve automatic operation, which can effectively improve work efficiency.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to information control methods, devices, electronic devices, and storage media. Background Technology

[0002] Currently, when controlling equipment to perform operations, operators need to perform many repetitive tasks. For example, when controlling equipment to perform transfer operations, operators need to repeatedly control the equipment to pick up the items and then control the equipment to place the items into the designated space.

[0003] Each transfer operation requires manual adjustment of the equipment's posture, which is complex and demands a high level of skill from the operator, resulting in low efficiency in controlling the equipment for transfer operations. Summary of the Invention

[0004] This application provides information control methods, apparatus, electronic devices, and storage media, which can improve the efficiency of controlling operating equipment to perform transfer operations.

[0005] This application provides an information control method, the method comprising:

[0006] Acquire the operation trajectory information of the operating equipment, the operation trajectory information including the operation posture and relative position relationship, the relative position relationship being the position relationship of the operating equipment relative to a specified space when the pick-up object is dropped;

[0007] In response to a selection operation performed on a scene image, a target point is determined, the scene image including the operating environment of the work equipment;

[0008] The working device is controlled to perform a picking operation at a first position, where the first position is the three-dimensional coordinate position corresponding to the target point;

[0009] The second position is calculated based on the target point cloud data and the preset point cloud model corresponding to the specified space. The target point cloud data is the point cloud data obtained by the working equipment when it is in the working pose.

[0010] The placement location is determined based on the second location and the relative positional relationship, and the operating equipment is controlled to perform a placement operation based on the placement location to place the picked-up item into the designated space.

[0011] This application embodiment also provides an information control device, the device comprising:

[0012] The acquisition module is used to acquire the operation trajectory information of the operating equipment. The operation trajectory information includes the operation posture and relative position relationship. The relative position relationship is the position relationship of the operating equipment relative to the specified space when the pick-up object is placed.

[0013] The determination module is used to determine the target point in response to a selection operation performed on the scene image;

[0014] The first control module is used to control the working equipment to perform a picking operation at a first position, where the first position is the three-dimensional coordinate position corresponding to the target point.

[0015] The calculation module is used to calculate the second position based on the target point cloud data and the preset point cloud model corresponding to the specified space. The target point cloud data is the point cloud data obtained by the working equipment in the working posture.

[0016] The second control module is used to determine the delivery location based on the second position and the relative positional relationship, and to control the operating equipment to perform a delivery operation based on the delivery location, so as to deliver the picked-up item into the designated space.

[0017] This application also provides an electronic device, including a memory storing multiple instructions; the processor loads instructions from the memory to execute steps in any of the information control methods provided in this application.

[0018] This application also provides a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute steps in any of the information control methods provided in this application.

[0019] This application embodiment can acquire the working pose and relative positional relationship; in response to a selection operation applied to the scene image, determine the target point; control the working device to perform a picking operation at the three-dimensional coordinate position corresponding to the target point; based on the target point cloud data acquired under the working pose and the preset point cloud model of the specified space, the second position can be accurately calculated; finally, based on the relative positional relationship and the target, the delivery position is calculated from the specified spatial position so that the picked item can be delivered to the specified space. When controlling the working device to perform a transfer operation, the user only needs to select the target point, which is simple and convenient and can effectively improve the efficiency of controlling the working device to perform a transfer operation. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram illustrating an application scenario of the information control method provided in the embodiments of this application;

[0022] Figure 2 This is a flowchart illustrating the information control method provided in an embodiment of this application;

[0023] Figure 3 This is a schematic diagram illustrating the generation of a virtual region provided in an embodiment of this application;

[0024] Figure 4 This is a schematic diagram of the coordinate system of each moving part in the working equipment provided in the embodiments of this application;

[0025] Figure 5 This is a flowchart illustrating an information control method provided in another embodiment of this application;

[0026] Figure 6 This is a schematic diagram of the structure of the information control device provided in the embodiments of this application;

[0027] Figure 7 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0028] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0029] This application provides information control methods, apparatus, electronic devices, and storage media.

[0030] Specifically, the information control device can be integrated into an electronic device, such as a terminal or server. The terminal can be a mobile phone, tablet, smart Bluetooth device, laptop, or personal computer (PC); the server can be a single server or a server cluster consisting of multiple servers.

[0031] In some embodiments, the information control device may also be integrated into multiple electronic devices. For example, the information control device may be integrated into multiple servers, and the information control method of this application may be implemented by multiple servers.

[0032] In some embodiments, the server may also be implemented as a terminal.

[0033] For example, refer to Figure 1The diagram illustrates an application scenario of the information control method provided in this embodiment. This scenario may include a display device 1000, an operating device 2000, an image acquisition device 3000, a point cloud acquisition device 4000, and a computing device 5000.

[0034] In this context, display device 1000 refers to any device with display functionality; operating equipment 2000 refers to mechanical equipment used for operations, such as excavators or intelligent robots; image acquisition device 3000 refers to equipment used for acquiring images, such as equipment with image acquisition functionality; point cloud acquisition device 4000 refers to equipment used for acquiring point cloud data, such as lidar. Computing device 5000 refers to any device with computing hardware capable of supporting and executing corresponding software products, such as the controller of operating equipment 2000.

[0035] The display device 1000, the work device 2000, the image acquisition device 3000, the point cloud acquisition device 4000, and the computing device 5000 can be connected via a network for data transmission.

[0036] The image acquisition device 3000 can acquire the working environment of the work equipment 2000 to obtain scene images, and transmit the acquired scene images to the display device 1000 for display. The image acquisition device 3000 can be installed on the work equipment 2000, or at a location that can acquire the working environment of the work equipment 2000, depending on the actual needs.

[0037] The point cloud acquisition device 4000 can collect corresponding point cloud data and transmit the collected point cloud data to the computing device 5000. The point cloud acquisition device 4000 can be installed on the operating device 2000 or at a location where the corresponding point cloud data can be collected, depending on the actual needs.

[0038] When executing the information control method, the computing device 5000 may acquire the operation trajectory information of the working device, which includes the working pose and relative positional relationship, wherein the relative positional relationship is the positional relationship of the working device relative to a specified space when tilting; determine a target point in response to a selection operation applied to a scene image; control the working device to perform a picking operation at a first position, wherein the first position is the three-dimensional coordinate position corresponding to the target point; calculate a second position based on a preset point cloud model of the specified space and target point cloud data; determine a placement position based on the second position and the relative positional relationship; and control the working device to perform a placement operation based on the placement position to place the picked-up object into the specified space.

[0039] The display device 1000, the work device 2000, and the computing device 5000 can be integrated into any two of them, or all of them can be integrated into one unit. For example, the display device 1000 and the work device 2000 can be integrated into one unit, while the computing device 5000 can be set up separately; or the display device 1000 and the computing device 5000 can be integrated into one unit, while the work device 2000 can be set up separately; or the work device 2000 and the computing device 5000 can be integrated into one unit, while the display device 1000 can be set up separately. The specific configuration method can be determined according to actual needs and is not specifically limited here.

[0040] In some implementations, if the point cloud acquisition device 4000 and the image acquisition device 3000 are installed on the work equipment 2000, in order to reduce the data transmission distance and achieve faster and more accurate control, the computing device 5000 and the work equipment 2000 can be integrated into one device. Thus, the computing device 5000 can quickly acquire relevant data and perform calculations to accurately control the work equipment 2000.

[0041] It should be noted that the remote control system of the operating equipment simulates the cockpit of the operating equipment and integrates video transmission, so that the operator can control the operating equipment in the simulated cockpit. This allows the operator to work in a comfortable environment while ensuring the operator's safety.

[0042] However, when controlling work equipment using a simulator, for tasks requiring multiple repetitive operations, such as transfer operations, the operator still needs to perform complex maneuvers within the simulator, combining transmitted images. This includes locating the pickup point, finding the designated space, and placing the pickup item into that space. This process is prone to misjudging the location of the designated space, leading to operational errors and reduced efficiency. Therefore, this application proposes an information control method. When controlling work equipment for transfer operations, the user only needs to select the target point, making the operation simple and convenient, and effectively improving operational efficiency. Detailed explanations follow.

[0043] In this embodiment, an information control method is provided, such as... Figure 2 As shown, the specific process of this information control method can be as follows:

[0044] S110. Obtain the operation trajectory information of the operating equipment, wherein the operation trajectory information includes the operation posture and relative position relationship.

[0045] Operation trajectory information refers to key information recorded in advance by manually controlled equipment during the transfer operation. The transfer operation refers to the process of controlling the equipment to grab and place an object into a designated space. For example, a transfer operation could be a loading operation; when excavating equipment is performing a loading operation, it first needs to excavate the object and then dump it into a vehicle.

[0046] Operating equipment refers to mechanical equipment that can grasp or drop objects. For example, operating equipment can be an excavator, whose bucket can grasp or drop objects. Operating equipment can also be an intelligent robot, whose robotic arm and robotic hand can work together to grasp or drop objects.

[0047] When controlling the work equipment to perform transfer operations, it is necessary to first control the work equipment to pick up the object at the location selected by the user, then move the work equipment to the vicinity of the designated space, and then adjust the robotic arm of the work equipment to be above the designated space in order to drop the object into the designated space.

[0048] In this context, "working pose" refers to the position and posture of the working equipment when it moves to the vicinity of a designated space. Working equipment typically includes multiple moving parts, and "working posture" refers to the posture of these moving parts. For example, when the working equipment is an excavator, its posture refers to the posture of its boom, arm, and bucket. Similarly, when the working equipment is an intelligent robot, its posture refers to the posture of each moving part that makes up the robot, such as the robotic arm or manipulator.

[0049] Relative positional relationship refers to the positional relationship of the working equipment relative to the designated space when the robotic arm of the working equipment is moved above the designated space, i.e., when dropping and picking up an object. For example, if the working equipment is an excavator, the relative positional relationship refers to the positional relationship of the excavator relative to the vehicle when the bucket of the excavator is moved above the truck bed of the vehicle.

[0050] The designated space refers to the space used to load the picked-up items. This designated space can be an area on the ground or a loading container, i.e., a container used to load the picked-up items, such as a vehicle or a box.

[0051] In some implementations, the operation trajectory information of the operating equipment can be pre-recorded and stored in a designated file, and the operation trajectory information can be directly obtained from the designated file.

[0052] In some implementations, the work trajectory information of the work equipment can be the data obtained by controlling the work equipment to perform transfer operations in real time.

[0053] S120, In response to a selection operation performed on the scene image, determine the target point.

[0054] The scene image includes the operating environment of the equipment, which can be obtained by capturing images of the operating environment of the equipment using an image acquisition device. After acquiring the scene image, the image acquisition device can send it to a display device for display to the user.

[0055] In some implementations, an image acquisition device may be installed on the work equipment. This image acquisition device can acquire images of the work environment of the work equipment to obtain scene images. Then, the display device can obtain the scene images from the image acquisition device and display the scene images to the user.

[0056] In some implementations, the display device is integrated into the electronic device, and when displaying a scene image to a user, the scene image may be displayed directly on the display of the electronic device. For example, the scene image may be displayed on a graphical user interface provided by the electronic device. When a user wants to control the work equipment to perform a transfer operation, the user can make a selection operation on the scene image, and the electronic device can determine the target point in response to the selection operation on the scene image.

[0057] In other embodiments, the display device and the electronic device can be two separate devices. For example, the display device is integrated into the work device. When the display device detects a selection operation on the scene image, it can transmit the corresponding data to the electronic device, so that the electronic device can determine the target point in response to the selection operation on the scene image.

[0058] The selection operation can include touch, swipe, tap, long press, short press, double tap, click, and end swipe. For example, in some embodiments, the selection operation can be a short press; in some embodiments, the selection operation can be a long press; and in some embodiments, the selection operation can be a swipe.

[0059] In some implementations, the selection operation can also be a combination of operations, such as a combination of click operations and click operations.

[0060] As one implementation method, a selection control corresponding to the operation can be displayed and selected on the graphical user interface. The user can control the selection control to move on the scene image through peripherals, such as a keyboard or mouse, so as to perform the selection operation.

[0061] The target point refers to the point selected by the selection operation. After the selection operation is detected, the point selected by the selection operation can be used as the target point.

[0062] S130. Control the working device to perform a picking operation at a first position, where the first position is the three-dimensional coordinate position corresponding to the target point.

[0063] The first position refers to the three-dimensional coordinate position corresponding to the target point. This three-dimensional coordinate position can refer to coordinate information in a three-dimensional coordinate system. The target point is a selected point in the scene image, and its position in the scene image is a two-dimensional coordinate position, that is, coordinate information in a two-dimensional coordinate system. Therefore, when controlling the working device to perform a picking operation at the first position, it can be done by acquiring the two-dimensional coordinates of the target point in the scene image; determining the three-dimensional coordinates of the target point in the camera coordinate system based on the two-dimensional coordinates and the currently acquired first point cloud data, where the camera coordinate system is the coordinate system corresponding to the device acquiring the scene image; determining the position indicated by the three-dimensional coordinates in the camera coordinate system as the first position; and controlling the working device to perform a picking operation at the first position.

[0064] When an image acquisition device captures scene images, it actually maps the spatial points of the object being photographed to a two-dimensional coordinate system to form a scene image. In other words, each point in the scene image corresponds to a spatial point in the real world. After obtaining the two-dimensional coordinates of the target point in the scene image, these two-dimensional coordinates can be transformed into the camera coordinate system to obtain the corresponding three-dimensional coordinates.

[0065] In some implementations, when obtaining the two-dimensional coordinates of the target point in the scene image, one can obtain the image coordinate system corresponding to the image acquisition device during imaging, and then determine the coordinates of the target point in that image coordinate system to obtain the two-dimensional coordinates of the target point.

[0066] Since the acquired scene image lacks depth information, the 3D coordinates of the target point cannot be directly determined using the intrinsic parameters of the image acquisition device. Therefore, the 3D coordinates of the target point can be determined by combining the point cloud data acquired by the point cloud acquisition device. Thus, in some embodiments, when determining the 3D coordinates of the target point in the camera coordinate system based on the 2D coordinates and the currently acquired first point cloud data, a virtual region can be generated by extending a preset number of pixels in a preset direction centered on the 2D coordinates; the points in the first point cloud data are projected onto the scene image to obtain the projection point corresponding to each point in the first point cloud data; and the 3D coordinates of the target point in the camera coordinate system are determined based on the first point cloud data in which the projection points fall within the virtual region.

[0067] Here, the center is a two-dimensional coordinate system, specifically the target point in the scene image. The preset direction refers to the front, back, left, and right sides of the target point. Expanding this preset direction by a preset number of pixels generates a virtual region. The preset number can be set according to actual needs; in this embodiment, a preset number of 10 is used for illustration. See also... Figure 3 The diagram illustrates the generation of a virtual region, where the target point is point A. By extending 10 pixels forward, backward, left, and right from point A, a virtual region 101 can be generated.

[0068] While acquiring scene images, electronic devices can also acquire point cloud data currently collected by point cloud acquisition devices. Point cloud data refers to a dataset of points in a certain coordinate system, which includes the three-dimensional coordinate information, color information, etc. of the points in that coordinate system. For ease of description, the point cloud data acquired here can be referred to as the first point cloud data.

[0069] Then, the points in the first point cloud data can be projected onto the scene image to obtain the projected point corresponding to each point in the first point cloud data. In some implementations, the camera intrinsic parameters of the image acquisition device and the relative extrinsic parameters of the image acquisition device and the point cloud acquisition device can be obtained first. Since the first point cloud data can contain the three-dimensional coordinates of each point in the coordinate system corresponding to the point cloud acquisition device, each point in the first point cloud data can be transformed to the camera coordinate system according to the following formula:

[0070]

[0071] Among them, P C This represents the coordinate information of a point in the camera coordinate system; This represents the relative extrinsic parameters of the image acquisition device and the point cloud acquisition device; P L This represents the coordinate information of a point in the coordinate system corresponding to the point cloud acquisition device.

[0072] Then, using the following formula, calculate the projection points from the points in the first point cloud data onto the scene image:

[0073]

[0074] Where Z represents point P in the camera coordinate system. C The distance to the camera's optical center; K represents the intrinsic parameters of the image acquisition device; (u,v) represents the two-dimensional coordinates of the point projected onto the scene image.

[0075] Through the above calculations, the two-dimensional coordinates of the projection point corresponding to each first point cloud data can be calculated. Then, based on the first point cloud data in the virtual area where the projection point falls, the three-dimensional coordinates of the target point in the camera coordinate system can be determined.

[0076] In some implementations, the two-dimensional coordinates corresponding to each projection point and the two-dimensional coordinates corresponding to the vertices of the virtual region can be obtained; the two-dimensional coordinates corresponding to the projection points and the two-dimensional coordinates corresponding to the vertices can be compared to determine the projection points that fall within the virtual rectangle.

[0077] As one implementation method, when calculating the two-dimensional coordinates of the vertices of the virtual region, one can obtain the distance corresponding to a single pixel; based on the distance corresponding to a single pixel and the two-dimensional coordinates of the target point, the two-dimensional coordinates of each vertex of the virtual region are calculated. For example, if the distance corresponding to a single pixel is 0.2 mm, the distance corresponding to 10 pixels is 2 mm, and the two-dimensional coordinates of the target point are (2, 4), then the two-dimensional coordinates corresponding to the four vertices of the virtual region are (0, 2), (0, 6), (4, 2), and (4, 6), respectively.

[0078] Then, based on the two-dimensional coordinates of each vertex, determine the first coordinate range on the first coordinate axis and the second coordinate range on the second coordinate axis. For example, if the x-axis is the first coordinate axis, the first coordinate range is 0 to 4; if the y-axis is the second coordinate axis, the second coordinate range is 2 to 6.

[0079] Then, the two-dimensional coordinates of each projection point are compared with the first coordinate range and the second coordinate range. The projection points whose coordinate values ​​on the first coordinate axis are within the first coordinate range and whose coordinate values ​​on the second coordinate axis are within the second coordinate range are determined as the projection points that fall into the virtual area.

[0080] Once the projection point falling into the virtual area is determined, the first point cloud data corresponding to that projection point can be obtained. In some embodiments, when using the first point cloud data of the projection point falling into the virtual area to determine the three-dimensional coordinates of the target point in the camera coordinate system, the first point cloud data of the projection point falling into the virtual area can be clustered according to Euclidean distance to obtain the corresponding clusters. If the number of clusters is a preset number, the coordinates of the cluster center of the cluster are used as the three-dimensional coordinates. If the number of clusters is greater than the preset number, the coordinates of the cluster center of the target cluster are used as the three-dimensional coordinates. The target cluster is the cluster with the most points.

[0081] Euclidean distance is the straight-line distance between two points in the first point cloud data. For ease of description, the first point cloud data in which the projected point falls into the virtual area is denoted as the first target point cloud data. When performing clustering processing on the first target point cloud data using Euclidean distance, any point can be selected from the first target point cloud data as the initial point; the Euclidean distance between the initial point and other points can be calculated, where other points refer to points in the first target point cloud data other than the initial point; points with Euclidean distances less than a preset value can be clustered into a cluster until no more points are added to the cluster.

[0082] In some implementations, to accelerate the clustering process, a KD-tree can be constructed first based on the first point cloud data where the projected points fall into the virtual region. A KD-tree, also known as a K-dimensional tree, is a spatially partitioned data structure. The first point cloud data is typically a 3D point cloud; therefore, the constructed KD-tree has a dimension of 3. Clustering is then performed based on the constructed KD-tree.

[0083] The clustering process described above yields the corresponding clusters, and the number of clusters can then be determined. If the number of clusters is the preset number, the coordinates of the cluster center can be directly used as the 3D coordinates of the target point. If the number of clusters is greater than the preset number, the cluster with the most points can be selected as the target cluster, and the coordinates of the target cluster center can then be used as the 3D coordinates of the target point.

[0084] In this embodiment, the preset number is 1. That is, when there is only one cluster, the three-dimensional coordinates corresponding to the cluster center of the cluster can be directly calculated to obtain the three-dimensional coordinates corresponding to the target point. When there are more than one cluster, the cluster with the most points can be taken as the target cluster, and the three-dimensional coordinates corresponding to the cluster center of the target cluster can be taken as the three-dimensional coordinates corresponding to the target point.

[0085] This allows the location indicated by the three-dimensional coordinates of the target point to be determined as the first position, and then the operating equipment can be controlled to perform the picking operation at the first position.

[0086] S140. Calculate the second position based on the target point cloud data and the preset point cloud model of the specified space.

[0087] After the control equipment performs the picking operation at the first position, it can be automatically controlled to adjust to the working pose. The target point cloud data refers to the point cloud data obtained by the point cloud acquisition device when the equipment is in the working pose.

[0088] The preset point cloud model for the specified space is obtained in advance. For example, before using the preset point cloud model for the specified space, spatial point cloud data corresponding to the specified space can be collected; the spatial point cloud data can be filtered to obtain spatial point cloud data to be clustered; and the spatial point cloud data to be clustered can be clustered based on Euclidean distance to obtain the preset point cloud model.

[0089] For example, point cloud data can be collected from all directions of a specified space using a point cloud acquisition device to obtain point cloud data for that space. This data can then be filtered, for example, by filtering out point clouds within a specified area or from the ground, resulting in point cloud data for the specified space to be clustered. Finally, Euclidean distance is used to cluster this data to obtain a pre-defined point cloud model for the specified space.

[0090] The second position refers to the actual position in the currently specified space. In some implementations, to accurately determine the second position, when calculating the second position based on the target point cloud data and a preset point cloud model of the specified space, the target point cloud data may be filtered to obtain point cloud data to be used; the point cloud data to be used may be clustered to obtain candidate clusters; the preset point cloud model may be matched with each candidate cluster to obtain matching parameters corresponding to each candidate cluster; based on the matching parameters, the candidate clusters that meet the preset matching conditions may be determined as clusters to be calculated; and the second position may be calculated using the clusters to be calculated.

[0091] After acquiring the target point cloud data, it can be filtered to obtain usable point cloud data. This filtering process can include discarding point cloud data within a specified range, filtering ground point clouds, etc.

[0092] After obtaining the point cloud data to be used, clustering processing can be performed on the point cloud data to obtain candidate clusters. Candidate clusters refer to the clusters obtained after clustering the point cloud data. In some implementations, Euclidean distance-based clustering can be used to obtain candidate clusters.

[0093] In some implementations, before performing clustering processing on the point cloud data to be used, it can be determined whether the point cloud data to be used meets the preset clustering conditions. For example, it can be to obtain the total number of points in the point cloud data to be used; determine the highest and lowest points in the preset direction from the point cloud data to be used; calculate the target distance between the highest and lowest points in the preset direction; if the total number is greater than the preset number and the target distance is greater than the preset distance, perform clustering processing on the point cloud data to be used to obtain the corresponding candidate clusters.

[0094] The preset direction refers to a pre-defined direction, which can be the direction indicated by a coordinate axis in the coordinate system corresponding to the point cloud acquisition device. For example, it can be the direction indicated by the z-axis in the coordinate system corresponding to the point cloud acquisition device.

[0095] Then, the highest and lowest points in the preset direction can be determined from the point cloud data to be used. In some implementations, the three-dimensional coordinates of each point are recorded in the point cloud data to be used. The maximum and minimum values ​​of these three-dimensional coordinates on the z-axis can be obtained, and the point corresponding to the maximum value is taken as the highest point, and the point corresponding to the minimum value is taken as the lowest point. Then, based on the maximum and minimum values, the target distance between the highest and lowest points in the preset direction can be calculated. For example, if the maximum value is 3 and the minimum value is 0, then the target distance = 3 - 0 = 3.

[0096] Then, based on the total number and the target distance, it is determined whether the point cloud data to be used meets the clustering conditions. For example, if the total number is greater than the preset number and the target distance is greater than the preset distance, the point cloud data to be used can be considered to meet the clustering conditions, and the subsequent clustering process can continue.

[0097] If the total number is less than or equal to the preset number, or the target distance is less than or equal to the preset distance, the point cloud data to be used is considered not to meet the clustering conditions, and subsequent clustering processing cannot be performed. In this case, a control signal can be generated to control the operating equipment to adjust its posture and reacquire the target point cloud data; this allows for continued filtering of the target point cloud data to obtain the point cloud data to be processed, until the point cloud data to be processed meets the clustering conditions. In this embodiment, the preset number can be 3000, and the preset distance can be 2m.

[0098] After clustering the point cloud data to be processed to obtain candidate clusters, a preset point cloud model in a specified space can be matched with each candidate cluster to obtain matching parameters for each candidate cluster. Matching the preset point cloud model with each candidate cluster involves transforming the candidate clusters to maximize their overlap with the preset point cloud model. The matching parameters represent the degree of overlap between the transformed candidate clusters' point clouds and the preset point cloud model.

[0099] When matching the preset point cloud model with each candidate cluster, the matching calculation can be performed using the Iterative Closest Point (ICP) algorithm. The ICP algorithm finds the corresponding point pairs between the preset point cloud model and the candidate cluster, constructs a rotation and translation matrix based on the corresponding point pairs, and uses the obtained matrix to transform the preset point cloud model into the coordinate system of the candidate cluster. It estimates the error function between the transformed preset point cloud model and the candidate cluster until the given error requirement is met, so as to obtain the matching parameters corresponding to each candidate cluster.

[0100] Then, the matching parameters corresponding to each candidate cluster are determined, and it is determined whether the candidate clusters meet the preset matching conditions. Candidate clusters that meet the preset matching conditions are then selected as the clusters to be calculated. Specifically, determining whether a candidate cluster meets the preset matching conditions can be done by checking if the matching parameter is less than a preset matching value. If the matching parameter of a candidate cluster is less than or equal to the preset matching value, the candidate cluster is considered to meet the preset matching conditions; if the matching parameter is greater than the preset matching value, the candidate cluster is considered not to meet the preset matching conditions. In some implementations, the matching parameter can refer to the average reprojection error; if the average reprojection error is less than 0.05m, the preset matching conditions are considered met.

[0101] The cluster to be calculated is a candidate cluster that meets the preset matching conditions. In other words, the preset point cloud model can overlap with the cluster to be calculated through transformations, such as rotation and translation. That is, the point cloud data in the cluster to be calculated is the point cloud data currently corresponding to the specified space.

[0102] Therefore, by using the cluster to be calculated, the actual location in a specified space can be calculated to obtain the second location. In some implementations, the three-dimensional coordinates of each point in the cluster to be calculated are also the three-dimensional coordinates of a point in a specified space. Based on the three-dimensional coordinates of each point in the cluster to be calculated, the location in the specified space can be calculated as the second location.

[0103] S150. Determine the placement location based on the second position and the relative positional relationship, and control the operating equipment to perform a placement operation based on the placement location to place the picked-up item into the designated space.

[0104] After calculating the second position, the delivery position can be calculated using the relative positional relationship in the operation trajectory information. The delivery position refers to the position of the robotic arm when the operating equipment delivers the picked-up object to the designated space.

[0105] In some implementations, when determining the placement location based on the second location and the relative positional relationship, and controlling the working equipment to perform placement operations based on the placement location, the placement location may be calculated using the second location and the relative positional relationship; the working equipment may be controlled to perform placement operations at the placement location to place the picked-up item into a designated space.

[0106] Since the relative positional relationship refers to the position of the working equipment relative to the designated space when dropping and picking up the object, and the position of the working equipment relative to the designated space specifically refers to the position of the robotic arm of the working equipment relative to the designated space, in some implementations, the dropping position can be calculated based on this relative positional relationship and the second position. For example, the relative positional relationship can be expressed as a function: y1 = αy2; where y1 represents the position of the working equipment, y2 represents the position of the designated space, and α represents other parameters. Then, by replacing y2 with the second position, the position of the corresponding working equipment, i.e., the dropping position, can be obtained.

[0107] This allows the robotic arm of the work equipment to be moved to the delivery location to perform the delivery operation, thereby delivering the picked-up item to the designated space.

[0108] The working equipment may include multiple moving parts, such as a robotic arm or robotic hand. To control the robotic arm's movement to the delivery location, these moving parts typically need to cooperate with each other. For example, when the working equipment is an excavator, the moving parts may include a boom, arm, cab, and bucket. To control the bucket's movement to the delivery location, multiple moving parts need to cooperate with each other. The designated space can be a designated loading container, which can refer to an object used to load the retrieved item, such as a vehicle.

[0109] In some implementations, the designated space can be a designated loading container, such as a vehicle or box. To ensure the safe movement of the robotic arm of the work equipment, when controlling the robotic arm of the work equipment to perform the delivery operation at the delivery location, the following steps can be taken: acquiring the posture parameters corresponding to each of the moving parts and the external dimensions of the designated loading container; generating a virtual object that matches the external dimensions; calculating the motion trajectory of the robotic arm to the delivery location based on the posture parameters corresponding to each moving part and the virtual object; and controlling the robotic arm to move to the delivery location based on the motion trajectory to perform the delivery operation.

[0110] The attitude parameters corresponding to the moving parts can include the angular velocity of each moving part and the angle between adjacent moving parts. An inertial measurement unit (IMU) can be installed on each moving part of the working equipment to obtain its attitude parameters such as angular velocity and angle.

[0111] For example, when the working equipment is an excavating device, the attitude parameters corresponding to the moving parts can include the angular velocities of the cab, boom, forearm, and bucket, as well as the angles between the cab and boom, the boom and forearm, and the forearm and bucket. IMUs can be installed on the excavating equipment, specifically on the cab, boom, forearm, and bucket.

[0112] The attitude parameters of moving parts can be defined based on the coordinate system corresponding to each moving part. For example, see [reference needed]. Figure 4 This diagram shows a schematic representation of the coordinate system of each moving part in the working equipment.

[0113] The operating equipment is an excavating device, and a total of 6 coordinate systems can be defined on the operating equipment, namely the coordinate system base_link of the operating equipment as a whole, the coordinate system upper_body_link of the upper body, the coordinate system arm_base_link of the base of the boom, the coordinate system boom_link of the boom, the coordinate system arm_link of the forearm, and the coordinate system gripper_link of the bucket.

[0114] Among them, the coordinate system arm_base_link at the base of the upper arm and the coordinate system upper_body_link of the upper body are fixed in transformation relationship, and the transformation relationship between the corresponding coordinate systems of other adjacent moving parts can be obtained in real time through the installed IMU.

[0115] As shown in the diagram, the coordinate systems of the boom, forearm, and bucket can rotate around the y-axis. Rotating downwards is counterclockwise, with the joint angles gradually increasing, while rotating downwards is clockwise, with the joint angles gradually decreasing. The coordinate system of the upper body component, upper_body_link, rotates around the z-axis. Rotating to the left is counterclockwise, with the joint angles gradually increasing, while rotating to the right is clockwise, with the joint angles gradually decreasing.

[0116] The joint zero position is defined as follows: the coordinate system arm_base_link at the base of the boom forms a 1-radian angle with the boom coordinate system boom_link; the coordinate system arm_link of the forearm forms a 1.5-radian angle with the boom coordinate system boom_link; and the coordinate system gripper_link of the bucket is parallel to the forearm coordinate system arm_link. If the forearm rotates upward by 1 radian at this point, the corresponding output joint angle of the forearm is -1 radian.

[0117] Therefore, by using the IMUs installed on each moving part, the corresponding attitude parameters can be obtained.

[0118] The external dimensions of a specified space refer to its overall length, width, and height. In some embodiments, these dimensions can be directly measured. In other embodiments, if the specified space is a vehicle, the length, width, and height of the vehicle's cab and the length, width, and height of the cargo bed can be measured. The external dimensions of the specified space can then be calculated based on these measurements.

[0119] After obtaining the external dimensions of the specified space, a virtual object matching those dimensions can be generated. Then, based on the posture parameters and the virtual object, the motion trajectory of the robot arm moving to the deployment position is calculated. Specifically, when calculating the motion trajectory of the robot arm moving to the deployment position, the current position of the robot arm can be used as the starting point and the deployment position as the ending point. By combining kinematic equations, the motion trajectory of the robot arm from the starting point to the ending point can be planned. Furthermore, obstacles in the space can be simulated based on the generated virtual object, and obstacle avoidance planning can be implemented based on the size and position of the virtual object to obtain the corresponding motion trajectory.

[0120] Then, based on the motion trajectory, the corresponding pose of the robot during the motion can be obtained. The values ​​of the variables of each joint angle can be directly calculated using the inverse kinematics equation. Based on the values ​​of the variables of each joint angle, the robot can be controlled to move to the delivery position according to the motion trajectory and perform the delivery operation.

[0121] In some implementations, the work trajectory information may also include an initial pickup location, which may be a pickup location actively selected manually during the acquisition of the loading trajectory, so that the work equipment can pick up the pickup at the pickup location.

[0122] After the control equipment performs a pickup operation at the first position, the initial pickup position can be updated using the first position. After the control equipment performs a delivery operation based on the delivery position, the robotic arm of the control equipment can be directly moved to the initial pickup position. It should be noted that since the initial pickup position has already been updated using the first position, the robotic arm actually moves to the first position after the control equipment performs the delivery operation.

[0123] It should be noted that in the aforementioned S130 to S150, no user intervention is required to automatically control the operating equipment to pick up and place items, thereby realizing the transfer operation.

[0124] In acquiring the work trajectory information, manual control of the work equipment is required for the transfer operation. When acquiring the work trajectory information of the work equipment, the following steps can be taken: In response to the trajectory recording operation, an initial pickup position is obtained; the work equipment is controlled to dig at the initial pickup position; in response to the matching trigger operation, specified point cloud data is acquired; if the preset point cloud model corresponding to the specified space and the specified point cloud data are successfully matched, the position of the specified space relative to the work equipment and the posture of the work equipment are recorded to obtain the work pose; based on the work pose, the center point corresponding to the specified space is calculated, and the work equipment is controlled to move to the center point; in response to the confirmation operation, the position of the center point relative to the work equipment is recorded to obtain the relative position relationship.

[0125] A trajectory recording operation refers to an operation that triggers trajectory recording. This trajectory recording operation can include touch, swipe, tap, long press, short press, double tap, click, and end swipe operation. For example, in some embodiments, the trajectory recording operation can be a short press operation; in some embodiments, the trajectory recording operation can be a long press operation; and in some embodiments, the trajectory recording operation can be a swipe operation.

[0126] As one implementation method, a corresponding recording control can be displayed in the graphical user interface showing the scene image. The user can click the recording control to automatically guide them in recording the work trajectory. For example, after the user clicks the recording control, a prompt message "Please select the location to pick up in the scene image" can be displayed. The user can then select a location in the scene image based on this prompt, allowing the electronic device to determine the target point and control the work equipment to perform the pickup operation at the first location indicated by the target point. Details regarding determining the target point and controlling the work equipment to perform the pickup operation at the first location can be found in the corresponding descriptions in S120 to S130 above, and will not be repeated here.

[0127] After the work equipment has finished picking up the data, it can respond to the matching trigger operation to obtain the specified point cloud data currently collected by the point cloud acquisition equipment. The matching trigger operation refers to the operation of triggering the matching calculation between the currently collected point cloud data and the preset point cloud model in the specified space.

[0128] In some implementations, after the picking is complete, a prompt message "Please adjust the work equipment to the specified space" can be displayed, along with a "Start Matching" control. This allows the user to manually adjust the work equipment to the specified spatial position and then click the "Start Matching" control to perform matching calculations.

[0129] When a matching trigger operation is detected, the point cloud data currently collected by the point cloud device can be acquired and recorded as the specified point cloud data. Then, matching calculations are performed based on the specified point cloud data and the preset point cloud model of the specified space. For details of the matching calculation, please refer to the corresponding description in S140 above, which will not be repeated here. If the preset point cloud model of the specified space and the specified point cloud data fail to match, it indicates that the candidate clusters obtained after clustering the specified point cloud data do not meet the preset matching conditions. The user can be prompted to readjust the posture of the operating device so that the matching calculation can be re-performed until the candidate clusters obtained after clustering the specified point cloud data meet the preset matching conditions, at which point the matching is considered successful.

[0130] Upon successful matching, the position of the specified space relative to the working equipment and the attitude of the working equipment are recorded to obtain the working pose. After successful matching, the position and boundary parameters of the specified space can be located. For example, when the specified space is a vehicle, the position of the vehicle, as well as the physical coordinates of the four corners of the vehicle bed and the four corners of the vehicle front, can be determined after successful matching. Then, using the physical coordinates of the four corners of the vehicle bed, the position of the corresponding center point of the bed can be calculated, and the robot arm of the working equipment can be controlled to move to the center point of the bed. The physical coordinates of the four corners of the vehicle front can be used as part of the vehicle's external dimensions and can be used in obstacle avoidance planning to ensure that the robot arm moves safely to the center point of the bed.

[0131] After the robotic arm of the control equipment moves to the center point of the designated space, a prompt message can continue to be displayed to prompt the user to confirm the position. For example, it could display a prompt message asking "Record this position?" and show the corresponding confirmation and denial controls.

[0132] The confirmation operation refers to confirming the current position of the working equipment. This confirmation operation can be a click operation on the confirmation control. After the user clicks the confirmation control, it is considered that the user has confirmed that the position is correct, and the working equipment can drop the pickup item. At the same time, the position of the center point relative to the working equipment can be recorded to obtain the relative position relationship. At this point, the work trajectory recording is considered complete, and the working equipment can automatically return to the initial pickup position after dropping the pickup item.

[0133] The information control method provided in this application can be applied to various automated operation scenarios. For example, taking the automatic loading of an excavator as an example, the solution provided in this application only requires the user to select a target point in the scene image, and the excavator can be automatically controlled to dig at the corresponding first position. After digging, the excavator will automatically find the designated space and place the picked-up object into the designated space.

[0134] The method provided in this application embodiment can acquire the working pose and relative position relationship; determine the target point in response to a selection operation on the scene image; control the working device to perform a picking operation at the three-dimensional coordinate position corresponding to the target point; accurately calculate the second position based on the target point cloud data acquired under the working pose and the preset point cloud model of the specified space; and finally calculate the delivery position from the specified space position based on the relative position relationship and the target, so as to dump the picked-up object into the specified space. When controlling the working device to perform a transfer operation, the user only needs to select the target point, which is simple and convenient. In the process of automatically executing the transfer operation, the second position can be automatically determined, and the working device can be controlled to complete autonomous obstacle avoidance and path planning, thereby improving the working efficiency when controlling the working device.

[0135] The method described in the above embodiments will be further described in detail below.

[0136] In this embodiment, loading operations using an excavator will be described in detail, with the designated space representing the vehicle. IMUs are installed on the excavator's boom, arm, bucket, and cab. A point cloud acquisition device, such as a lidar sensor, can be installed at the front of the excavator cab to collect scene images and generate laser point clouds. An image acquisition device, also installed at the front of the excavator cab, is used to acquire scene images to generate the scene displayed on the client interface. This image acquisition device can consist of multiple monocular cameras, such as three monocular cameras, which stitch the images together to generate the scene image. The electronic equipment executing the information control method can be integrated with the excavator, while the display device is separate from the excavator, enabling remote control of the excavator for loading operations. The method of this embodiment will be described in detail below.

[0137] like Figure 5 As shown, the specific process of an information control method is as follows:

[0138] S210: In response to the trajectory recording operation, acquire the excavator's working trajectory information.

[0139] The display device can show a recording control. When the user clicks the recording control, it will automatically guide the user to record the loading trajectory. The user can first select the location to be excavated (i.e., the target point) in the scene image. The three-dimensional coordinate position corresponding to the target point can be called the initial pick position and recorded. The excavator can then automatically start digging at the initial pick position.

[0140] After excavation is complete, the user can manually adjust the excavator's bucket to the vehicle's position. At this point, a "Start Matching" control will appear on the graphical user interface. Clicking "Start Matching" will cause the excavator to automatically perform matching calculations based on the currently collected point cloud data and the vehicle's preset point cloud data. Upon successful matching, the vehicle's position relative to the excavator and the excavator's attitude are recorded to obtain the working pose.

[0141] The vehicle's position can be obtained during the matching calculation. Using the vehicle's position and dimensions, the center point of the vehicle's bucket can be calculated. Then, the excavator's bucket is automatically adjusted to the center point. After user confirmation, the relative positional relationship is obtained, that is, the relative position of the center points of the excavator's bucket and the vehicle's bucket when dumping excavated materials.

[0142] At this point, the operation trajectory information can be obtained, namely the initial pickup position, operation pose, and relative positional relationship.

[0143] S220, in response to a selection operation performed on the scene image, determine the target point.

[0144] S230. Control the excavator to perform a pickup operation at a first position, where the first position is the three-dimensional coordinate position corresponding to the target point.

[0145] S240. Calculate the second position based on the target point cloud data and the vehicle's preset point cloud model.

[0146] S250. Determine the placement position based on the second position and the relative positional relationship, and control the excavator to perform a placement operation based on the placement position to dump the excavated material into the truck bed of the vehicle.

[0147] After obtaining the work trajectory information, the user can reselect the target point on the scene image. The location of the selected target point in the actual work environment can be recorded as the first position. The excavator can be automatically controlled to dig at the first position, realizing the function of digging wherever the target is pointed. At the same time, the initial pickup position is updated with the first position.

[0148] Subsequently, the excavator automatically adjusts to a loading posture and performs matching calculations to determine the second position, i.e., the actual position of the vehicle. Simultaneously, the recorded working pose is updated to the currently successfully matched working pose for subsequent accelerated matching calculations. Then, based on the vehicle's actual position and relative positional relationship, the delivery position is calculated, and the excavator's bucket is automatically moved to the delivery position to dump the excavated material into the vehicle's bucket. After dumping, the excavator automatically moves back to the initial pickup position to await the user's reselection of the target point.

[0149] During the movement of the excavator, there is no physical movement of the excavator itself; only the rotation of the cab and the movement of the boom, arm, and bucket are involved. During the movement, obstacles can be simulated based on the vehicle's size information to perform obstacle avoidance planning, thereby ensuring the safe movement of the excavator.

[0150] As shown above, by pre-acquiring the working pose and relative positional relationship, the user only needs to reselect the target point to automatically control the excavator to dig at the corresponding location. After digging, the vehicle position can be quickly located using the working pose, and the placement position can be obtained using the relative positional relationship. Then, the bucket is moved to the placement position to dump the excavated material into the vehicle's bucket. The entire process only requires the user to select the target point in the scene image to achieve automatic loading, which can effectively improve work efficiency. Furthermore, by setting the display device and excavator separately, the user can directly select the target point in the scene image presented on the display device to remotely control the excavator to perform loading operations, which can also ensure the operator's personal safety.

[0151] To better implement the above methods, this application also provides an information control device, which can be integrated into an electronic device, such as a terminal or server. The terminal can be a mobile phone, tablet computer, smart Bluetooth device, laptop computer, or personal computer; the server can be a single server or a server cluster composed of multiple servers.

[0152] For example, in this embodiment, the method of this application embodiment will be described in detail by taking the information control device specifically integrated into the terminal as an example.

[0153] For example, such as Figure 6 As shown, the information control device 300 may include an acquisition module 310, a determination module 320, a first control module 330, a calculation module 340, and a second control module 350.

[0154] The acquisition module 310 is used to acquire the operation trajectory information of the operating equipment. The operation trajectory information includes the operation posture and relative position relationship. The relative position relationship is the position relationship of the operating equipment relative to the specified space when the pick-up object is placed.

[0155] The determination module 320 is configured to determine a target point in response to a selection operation performed on a scene image, wherein the scene image includes the operating environment of the work equipment.

[0156] The first control module 330 is used to control the working device to perform a picking operation at a first position, where the first position is the three-dimensional coordinate position corresponding to the target point.

[0157] The calculation module 340 is used to calculate the second position based on the target point cloud data and the preset point cloud model of the specified space, wherein the target point cloud data is the point cloud data obtained by the working equipment when it is in the working pose;

[0158] The second control module 350 is used to determine the delivery position based on the second position and the relative positional relationship, and to control the operating equipment to perform a delivery operation based on the delivery position, so as to dump the picked-up item into the designated space.

[0159] In some embodiments, the first control module 330 further includes:

[0160] A two-dimensional acquisition unit is used to acquire the two-dimensional coordinates of the target point in the scene image;

[0161] The determining unit is used to determine the three-dimensional coordinates of the target point in the camera coordinate system based on the two-dimensional coordinates and the currently acquired first point cloud data, wherein the camera coordinate system is the coordinate system corresponding to the image acquisition device;

[0162] The target determination unit is used to determine the position indicated by the three-dimensional coordinates in the camera coordinate system as the first position;

[0163] A first control unit is used to control the working equipment to perform a picking operation at the first position.

[0164] In some embodiments, the determining unit is further configured to:

[0165] Using the two-dimensional coordinates as the center, a preset number of pixels are extended in a preset direction to generate a virtual region;

[0166] Projecting the points in the first point cloud data onto the scene image yields the projection point corresponding to each point in the first point cloud data.

[0167] Based on the first point cloud data where the projection point falls into the virtual area, the three-dimensional coordinates of the target point in the camera coordinate system are determined.

[0168] In some embodiments, the determining unit is further configured to:

[0169] Based on the Euclidean distance, the first point cloud data of the projection point falling into the virtual region is clustered to obtain the corresponding clusters;

[0170] If the number of clusters is a preset number, the coordinates of the cluster center of the clusters are used as the three-dimensional coordinates;

[0171] If the number of clusters is greater than a preset number, the coordinates of the cluster center of the target cluster are used as the three-dimensional coordinates, and the target cluster is the cluster with the most points.

[0172] In some embodiments, the computing module 340 further includes:

[0173] The filtering unit is used to filter the target point cloud data to obtain the point cloud data to be used.

[0174] Clustering unit, used to perform clustering processing on the point cloud data to be used to obtain candidate clusters;

[0175] A matching unit is used to match the preset point cloud model with each of the candidate clusters to obtain matching parameters corresponding to each candidate cluster;

[0176] The condition unit is used to determine the candidate clusters that meet the preset matching conditions as the clusters to be calculated based on the matching parameters;

[0177] The calculation unit is used to calculate the second position using the cluster to be calculated.

[0178] In some embodiments, before performing clustering processing on the point cloud data to be used to obtain candidate clusters, the clustering unit is further configured to:

[0179] Obtain the total number of points in the point cloud data to be used;

[0180] From the point cloud data to be used, determine the highest and lowest points in the preset direction;

[0181] Calculate the target distance between the highest point and the lowest point in the preset direction;

[0182] If the total number is greater than the preset number and the target distance is greater than the preset distance, the point cloud data to be used is clustered to obtain the corresponding candidate clusters.

[0183] In some embodiments, the second control module 350 further includes:

[0184] The calculation unit is used to calculate the delivery position using the second position and the relative positional relationship;

[0185] The second control unit is used to control the working equipment to perform a delivery operation at the delivery location to deliver the picked-up item into the designated space.

[0186] In some embodiments, the working device includes a plurality of movable parts, the plurality of movable parts including a robotic arm, the designated space being a designated loading container, and the second control unit is further configured to:

[0187] Obtain the attitude parameters corresponding to each of the active components, as well as the external dimensions of the specified loading container;

[0188] Generate a virtual object that matches the stated dimensions;

[0189] Based on the posture parameters corresponding to each active component and the virtual object, the motion trajectory of the robotic arm moving to the second position is calculated;

[0190] Based on the motion trajectory, the robotic arm is controlled to move to the delivery location and perform the delivery operation.

[0191] In some embodiments, the work trajectory information further includes an initial pickup position, and after controlling the work device to perform the pickup operation at the first position, the first control module 330 is further configured to:

[0192] Update the initial pickup position with the first position;

[0193] After determining the delivery location based on the second position and the relative positional relationship, and controlling the operating equipment to perform the delivery operation based on the delivery location, the second control module 350 is further configured to:

[0194] Control the working equipment to move to the initial pickup position.

[0195] In some embodiments, the acquisition module 310 is further configured to:

[0196] In response to the trajectory recording operation, the initial pickup position is obtained;

[0197] Control the working equipment to perform a picking operation at the initial picking position;

[0198] In response to a matching trigger operation, acquire the specified point cloud data;

[0199] If the preset point cloud model corresponding to the specified space and the specified point cloud data are successfully matched, the position of the specified space relative to the working equipment and the attitude of the working equipment are recorded to obtain the working pose;

[0200] Based on the work pose, calculate the center point corresponding to the specified space, and control the work equipment to move to the center point;

[0201] In response to the confirmation operation, the position of the center point relative to the working equipment is recorded to obtain the relative positional relationship.

[0202] In some embodiments, before calculating the second position based on the target point cloud data and the preset point cloud model of the specified space, the calculation module 340 is further configured to:

[0203] Collect spatial point cloud data corresponding to the specified space;

[0204] The spatial point cloud data is filtered to obtain spatial point cloud data to be clustered;

[0205] Based on Euclidean distance, the spatial point cloud data to be clustered is clustered to obtain the preset point cloud model.

[0206] In practice, each of the above modules or units can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above modules or units, please refer to the previous method embodiments, which will not be repeated here.

[0207] As can be seen from the above, the information control device in this embodiment can acquire the working pose and relative position relationship; in response to the selection operation acting on the scene image, determine the target point; control the working equipment to perform the picking operation at the three-dimensional coordinate position corresponding to the target point; based on the target point cloud data acquired under the working pose and the preset point cloud model of the specified space, the second position can be accurately calculated; finally, based on the relative position relationship and the target, the delivery position is calculated from the specified space position so that the picked item can be dumped into the truck bed of the specified space to realize the loading operation. The whole process only requires the user to select the target point, and the rest of the process can be executed automatically. The loading operation can be carried out accurately without user intervention, which can effectively improve the work efficiency.

[0208] Accordingly, this application also provides an electronic device, which can be a terminal or a server. The terminal can be a smartphone, tablet computer, laptop computer, touch screen, personal computer, personal digital assistant (PDA), or other electronic devices.

[0209] like Figure 7 As shown, Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 400 includes a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, and a computer program stored in the memory 402 and executable on the processor. The processor 401 and the memory 402 are electrically connected. Those skilled in the art will understand that the electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0210] The processor 401 is the control center of the electronic device 400. It connects various parts of the electronic device 400 through various interfaces and lines. By running or loading software programs and / or modules stored in the memory 402, and calling data stored in the memory 402, it performs various functions of the electronic device 400 and processes data, thereby monitoring the electronic device 400 as a whole.

[0211] In this embodiment, the processor 401 in the electronic device 400 loads the instructions corresponding to the processes of one or more applications into the memory 402 according to the following steps, and the processor 401 runs the applications stored in the memory 402 to realize various functions:

[0212] The system acquires the operation trajectory information of the working equipment, which includes the working pose and relative positional relationship, wherein the relative positional relationship is the positional relationship of the working equipment relative to a specified space when dropping the picked-up object; in response to a selection operation applied to a scene image, a target point is determined, wherein the scene image includes the working environment of the working equipment; the system controls the working equipment to perform a picking operation at a first position, wherein the first position is the three-dimensional coordinate position corresponding to the target point; a second position is calculated based on the target point cloud data and a preset point cloud model corresponding to the specified space, wherein the target point cloud data is the point cloud data acquired by the working equipment at the working pose; a drop position is determined based on the second position and the relative positional relationship, and the system controls the working equipment to perform a drop operation based on the drop position to drop the picked-up object into the specified space.

[0213] When operating equipment remotely, users only need to select the target point, making the operation simple and convenient, and effectively improving the efficiency of remotely controlled operations.

[0214] The two-dimensional coordinates of the target point in the scene image are obtained; based on the two-dimensional coordinates and the currently obtained first point cloud data, the three-dimensional coordinates of the target point in the camera coordinate system are determined, wherein the camera coordinate system is the coordinate system corresponding to the device that acquires the scene image; the position indicated by the three-dimensional coordinates in the camera coordinate system is determined as the first position; and the working device is controlled to perform a picking operation at the first position.

[0215] A virtual region is generated by extending a preset number of pixels in a preset direction around the two-dimensional coordinates; the points in the first point cloud data are projected onto the scene image to obtain the projection point corresponding to each point in the first point cloud data; the three-dimensional coordinates of the target point in the camera coordinate system are determined based on the first point cloud data in which the projection point falls into the virtual region.

[0216] Based on the Euclidean distance, the first point cloud data of the projection points falling into the virtual region are clustered to obtain the corresponding clusters; if the number of clusters is a preset number, the cluster center coordinates of the clusters are used as the three-dimensional coordinates; if the number of clusters is greater than the preset number, the cluster center coordinates of the target cluster are used as the three-dimensional coordinates, and the target cluster is the cluster with the most points.

[0217] By constructing a virtual region based on the selected target point and performing clustering processing on the first point cloud data of the projection point in the virtual region, the first position corresponding to the target point can be accurately obtained.

[0218] The target point cloud data is filtered to obtain point cloud data to be used; the point cloud data to be used is clustered to obtain candidate clusters; the preset point cloud model is matched with each candidate cluster to obtain matching parameters corresponding to each candidate cluster; based on the matching parameters, the candidate clusters that meet the preset matching conditions are determined as clusters to be calculated; the second position is calculated using the clusters to be calculated.

[0219] Obtain the total number of points in the point cloud data to be used; determine the highest and lowest points in a preset direction from the point cloud data to be used; calculate the target distance between the highest and lowest points in the preset direction; if the total number is greater than the preset number and the target distance is greater than the preset distance, perform clustering processing on the point cloud data to be used to obtain the corresponding candidate clusters.

[0220] During the operation, point cloud data can be used for matching to accurately determine the second position, thereby improving the accuracy of the operation.

[0221] Using the second position and the relative positional relationship, the placement position is calculated; the operating equipment is controlled to perform a placement operation at the placement position to dump the picked-up item into a designated space.

[0222] By utilizing the determined second position and relative positional relationship, the placement position can be accurately calculated, thus enabling the pick-up item to be accurately placed into the designated space.

[0223] Obtain the attitude parameters corresponding to each of the active components and the outer dimensions of the specified loading container; generate a virtual object that matches the outer dimensions; calculate the motion trajectory of the bucket moving to the delivery position based on the attitude parameters corresponding to each active component and the virtual object; control the bucket to move to the delivery position based on the movement path and execute the delivery operation.

[0224] When controlling the movement of the work equipment, a virtual object is generated based on the external dimensions of a specified loading container in a specified space. This facilitates obstacle avoidance during the calculation of the movement trajectory, which can effectively improve the safety of the operation.

[0225] The method further includes: updating the initial pickup position with the first position; determining the delivery position based on the second position and the relative position relationship, and controlling the operating equipment to perform delivery operations based on the delivery position; and controlling the operating equipment to move to the initial pickup position.

[0226] In response to the trajectory recording operation, an initial pickup position is obtained; the operating device is controlled to perform pickup operations at the initial pickup position; in response to the matching trigger operation, specified point cloud data is obtained; if the preset point cloud model corresponding to the specified space and the specified point cloud data are successfully matched, the position of the specified space relative to the operating device and the attitude of the operating device are recorded to obtain the operating pose; based on the operating pose, the center point corresponding to the specified space is calculated, and the operating device is controlled to move to the center point; in response to the confirmation operation, the position of the center point relative to the operating device is recorded to obtain the relative position relationship.

[0227] Collect spatial point cloud data corresponding to the specified space; filter the spatial point cloud data to obtain spatial point cloud data to be clustered; perform clustering processing on the spatial point cloud data to be clustered based on Euclidean distance to obtain the preset point cloud model.

[0228] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0229] Optional, such as Figure 7 As shown, the electronic device 400 also includes: a display screen 403, a radio frequency circuit 404, an audio circuit 405, an input unit 406, and a power supply 407. The processor 401 is electrically connected to the display screen 403, the radio frequency circuit 404, the audio circuit 405, the input unit 406, and the power supply 407. Those skilled in the art will understand that... Figure 7 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0230] Display screen 403 can be used to display a graphical user interface. Display screen 403 may include a general-purpose display screen and a touch display screen. The general-purpose display screen only has display functions, while the touch display screen can be used to display the graphical user interface and receive operation commands generated by the user interacting with the graphical user interface.

[0231] A touch display screen may include a display panel and a touch panel. The display panel can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces (GUIs) of electronic devices. These GUIs can be composed of graphics, text, icons, video, and any combination thereof. Optionally, the display panel can be configured using a liquid crystal display (LCD), organic light-emitting diode (OLED), or other similar technologies. The touch panel can collect user touch operations on or near the panel (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel), generate corresponding operation commands, and execute the corresponding program. Optionally, the touch panel may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch location and the signal generated by the touch operation, transmitting the signal to the touch controller. The touch controller receives touch information from the touch detection device, converts it into touch point coordinates, sends it to the processor 401, and can also receive and execute commands from the processor 401. The touch panel can cover the display panel. When the touch panel detects a touch operation on or near it, it transmits the information to the processor 401 to determine the type of touch event. Subsequently, the processor 401 provides corresponding visual output on the display panel based on the type of touch event. In this embodiment, the touch panel and the display panel can be integrated into a touch display screen to achieve input and output functions. However, in some embodiments, the touch panel and the display panel can be implemented as two independent components to achieve input and output functions. That is, the touch display screen can also be used as part of the input unit 406 to achieve input functions.

[0232] In this embodiment of the application, a control application is executed by processor 401 to generate a graphical user interface on display screen 403, which may include scene images.

[0233] The radio frequency circuit 404 can be used to transmit and receive radio frequency signals to establish wireless communication with network devices or other electronic devices, and to transmit and receive signals with network devices or other electronic devices.

[0234] Audio circuit 405 can be used to provide an audio interface between a user and an electronic device via a speaker and a microphone. Audio circuit 405 can convert received audio data into electrical signals and transmit them to the speaker, where the speaker converts them into sound signals for output. Conversely, the microphone converts collected sound signals into electrical signals, which are then received by audio circuit 405, converted back into audio data, and then processed by processor 401 before being transmitted via radio frequency circuit 404 to, for example, another electronic device, or output to memory 402 for further processing. Audio circuit 405 may also include an earphone jack to provide communication between peripheral headphones and electronic devices.

[0235] The input unit 406 can be used to receive input numbers, characters, or user characteristic information (such as fingerprints, iris, facial information, etc.), and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control.

[0236] Power supply 407 is used to supply power to various components of electronic device 400. Optionally, power supply 407 can be logically connected to processor 401 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. Power supply 407 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0237] although Figure 7 As not shown in the diagram, the electronic device 400 may also include a camera, sensor, wireless fidelity module, Bluetooth module, etc., which will not be described in detail here.

[0238] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0239] As can be seen from the above, the electronic device provided in this embodiment can acquire the working pose and relative position relationship; in response to the selection operation acting on the scene image, it determines the target point; it controls the working device to perform a picking operation at the three-dimensional coordinate position corresponding to the target point; based on the target point cloud data acquired under the working pose and the preset point cloud model of the specified space, it can accurately calculate the second position; finally, based on the relative position relationship and the target, it calculates the delivery position from the specified spatial position so that the picked-up item can be dumped into the truck bed of the specified space. When remotely controlling the working device to perform loading operations, the user only needs to select the target point, which is simple and convenient and can effectively improve the efficiency of remotely controlling the working device to perform loading operations.

[0240] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0241] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of computer programs that can be loaded by a processor to execute steps in any of the information control methods provided in embodiments of this application. For example, the computer program can execute the following steps:

[0242] The system acquires the operation trajectory information of the working equipment, which includes the working pose and relative positional relationship, wherein the relative positional relationship is the positional relationship of the working equipment relative to a specified space when dropping the picked-up object; in response to a selection operation applied to a scene image, a target point is determined, wherein the scene image includes the working environment of the working equipment; the system controls the working equipment to perform a picking operation at a first position, wherein the first position is the three-dimensional coordinate position corresponding to the target point; a second position is calculated based on the target point cloud data and a preset point cloud model corresponding to the specified space, wherein the target point cloud data is the point cloud data acquired by the working equipment at the working pose; a drop position is determined based on the second position and the relative positional relationship, and the system controls the working equipment to perform a drop operation based on the drop position to drop the picked-up object into the specified space.

[0243] When operating equipment remotely, users only need to select the target point, making the operation simple and convenient, and effectively improving the efficiency of remotely controlled operations.

[0244] The two-dimensional coordinates of the target point in the scene image are obtained; based on the two-dimensional coordinates and the currently obtained first point cloud data, the three-dimensional coordinates of the target point in the camera coordinate system are determined, wherein the camera coordinate system is the coordinate system corresponding to the device that acquires the scene image; the position indicated by the three-dimensional coordinates in the camera coordinate system is determined as the first position; and the working device is controlled to perform a picking operation at the first position.

[0245] A virtual region is generated by extending a preset number of pixels in a preset direction around the two-dimensional coordinates; the points in the first point cloud data are projected onto the scene image to obtain the projection point corresponding to each point in the first point cloud data; the three-dimensional coordinates of the target point in the camera coordinate system are determined based on the first point cloud data in which the projection point falls into the virtual region.

[0246] Based on the Euclidean distance, the first point cloud data of the projection points falling into the virtual region are clustered to obtain the corresponding clusters; if the number of clusters is a preset number, the cluster center coordinates of the clusters are used as the three-dimensional coordinates; if the number of clusters is greater than the preset number, the cluster center coordinates of the target cluster are used as the three-dimensional coordinates, and the target cluster is the cluster with the most points.

[0247] By constructing a virtual region based on the selected target point and performing clustering processing on the first point cloud data of the projection point in the virtual region, the first position corresponding to the target point can be accurately obtained.

[0248] The target point cloud data is filtered to obtain point cloud data to be used; the point cloud data to be used is clustered to obtain candidate clusters; the preset point cloud model is matched with each candidate cluster to obtain matching parameters corresponding to each candidate cluster; based on the matching parameters, the candidate clusters that meet the preset matching conditions are determined as clusters to be calculated; the second position is calculated using the clusters to be calculated.

[0249] Obtain the total number of points in the point cloud data to be used; determine the highest and lowest points in a preset direction from the point cloud data to be used; calculate the target distance between the highest and lowest points in the preset direction; if the total number is greater than the preset number and the target distance is greater than the preset distance, perform clustering processing on the point cloud data to be used to obtain the corresponding candidate clusters.

[0250] During the operation, point cloud data can be used for matching to accurately determine the second position, thereby improving the accuracy of the operation.

[0251] Using the second position and the relative positional relationship, the placement position is calculated; the operating equipment is controlled to perform a placement operation at the placement position to dump the picked-up item into a designated space.

[0252] By utilizing the determined second position and relative positional relationship, the placement position can be accurately calculated, thus enabling the pick-up item to be accurately placed into the designated space.

[0253] Obtain the attitude parameters corresponding to each of the active components and the outer dimensions of the specified loading container; generate a virtual object that matches the outer dimensions; calculate the motion trajectory of the bucket moving to the delivery position based on the attitude parameters corresponding to each active component and the virtual object; control the bucket to move to the delivery position based on the movement path and execute the delivery operation.

[0254] When controlling the movement of the work equipment, a virtual object is generated based on the external dimensions of a specified loading container in a specified space. This facilitates obstacle avoidance during the calculation of the movement trajectory, which can effectively improve the safety of the operation.

[0255] The method further includes: updating the initial pickup position with the first position; determining the delivery position based on the second position and the relative position relationship, and controlling the operating equipment to perform delivery operations based on the delivery position; and controlling the operating equipment to move to the initial pickup position.

[0256] In response to the trajectory recording operation, an initial pickup position is obtained; the operating device is controlled to perform pickup operations at the initial pickup position; in response to the matching trigger operation, specified point cloud data is obtained; if the preset point cloud model corresponding to the specified space and the specified point cloud data are successfully matched, the position of the specified space relative to the operating device and the attitude of the operating device are recorded to obtain the operating pose; based on the operating pose, the center point corresponding to the specified space is calculated, and the operating device is controlled to move to the center point; in response to the confirmation operation, the position of the center point relative to the operating device is recorded to obtain the relative position relationship.

[0257] Collect spatial point cloud data corresponding to the specified space; filter the spatial point cloud data to obtain spatial point cloud data to be clustered; perform clustering processing on the spatial point cloud data to be clustered based on Euclidean distance to obtain the preset point cloud model.

[0258] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0259] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0260] Since the computer program stored in the storage medium can execute the steps of any of the information control methods provided in the embodiments of this application, the beneficial effects that any of the information control methods provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0261] The above provides a detailed description of an information control method, apparatus, electronic device, and storage medium provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An information control method, characterized in that, The method includes: In response to the trajectory recording operation, the initial pickup position is obtained; The control equipment performs a pickup operation at the initial pickup position; In response to a matching trigger operation, acquire the specified point cloud data; If the preset point cloud model corresponding to the specified space and the specified point cloud data are successfully matched, the position of the specified space relative to the working device and the attitude of the working device are recorded to obtain the working pose; wherein, the specified space is the space for loading the picked-up object; Based on the work pose, calculate the center point corresponding to the specified space, and control the work equipment to move to the center point; In response to the confirmation operation, the position of the center point relative to the working equipment is recorded to obtain the relative positional relationship, thereby acquiring the working trajectory information of the working equipment; In response to a selection operation performed on a scene image, a target point is determined, the scene image including the operating environment of the work equipment; The working device is controlled to perform a picking operation at a first position, where the first position is the three-dimensional coordinate position corresponding to the target point; The second position is calculated based on the target point cloud data and the preset point cloud model corresponding to the specified space. The target point cloud data is the point cloud data obtained by the working equipment when it is in the working pose. The placement location is determined based on the second location and the relative positional relationship, and the operating equipment is controlled to perform a placement operation based on the placement location to place the picked-up item into the designated space.

2. The method according to claim 1, characterized in that, The control of the working device to perform a pickup operation at the first position includes: Obtain the two-dimensional coordinates of the target point in the scene image; Based on the two-dimensional coordinates and the currently acquired first point cloud data, the three-dimensional coordinates of the target point in the camera coordinate system are determined, wherein the camera coordinate system is the coordinate system corresponding to the device that acquires the scene image; The position indicated by the three-dimensional coordinates in the camera coordinate system is determined as the first position; Control the work equipment to perform a pickup operation at the first position.

3. The method according to claim 2, characterized in that, Determining the three-dimensional coordinates of the target point in the camera coordinate system based on the two-dimensional coordinates and the currently acquired first point cloud data includes: Using the two-dimensional coordinates as the center, a preset number of pixels are extended in a preset direction to generate a virtual region; Projecting the points in the first point cloud data onto the scene image yields the projection point corresponding to each point in the first point cloud data. Based on the first point cloud data where the projection point falls into the virtual area, the three-dimensional coordinates of the target point in the camera coordinate system are determined.

4. The method according to claim 3, characterized in that, Determining the three-dimensional coordinates of the target point in the camera coordinate system based on the first point cloud data where the projection point falls into the virtual region includes: Based on the Euclidean distance, the first point cloud data of the projection point falling into the virtual region is clustered to obtain the corresponding clusters; If the number of clusters is a preset number, the coordinates of the cluster center of the clusters are used as the three-dimensional coordinates; If the number of clusters is greater than a preset number, the coordinates of the cluster center of the target cluster are used as the three-dimensional coordinates, and the target cluster is the cluster with the most points.

5. The method according to claim 1, characterized in that, The step of calculating the second position based on the target point cloud data and the preset point cloud model of the specified space includes: The target point cloud data is filtered to obtain the point cloud data to be used; The point cloud data to be used is subjected to clustering processing to obtain candidate clusters; The preset point cloud model is matched with each of the candidate clusters to obtain the matching parameters corresponding to each candidate cluster; Based on the matching parameters, the candidate clusters that meet the preset matching conditions are determined as the clusters to be calculated; The second position is calculated using the cluster to be calculated.

6. The method according to claim 5, characterized in that, Before performing clustering processing on the point cloud data to be used to obtain candidate clusters, the process includes: Obtain the total number of points in the point cloud data to be used; From the point cloud data to be used, determine the highest and lowest points in the preset direction; Calculate the target distance between the highest point and the lowest point in the preset direction; If the total number is greater than the preset number and the target distance is greater than the preset distance, the point cloud data to be used is clustered to obtain the corresponding candidate clusters.

7. The method according to claim 1, characterized in that, The step of determining the delivery location based on the second location and the relative positional relationship, and controlling the operating equipment to perform the delivery operation based on the delivery location, includes: Calculate the delivery location using the second location and the relative positional relationship; The operating equipment is controlled to perform a delivery operation at the delivery location to deliver the picked-up item into a designated space.

8. The method according to claim 7, characterized in that, The operating equipment includes multiple movable parts, including a robotic arm. The designated space is a designated loading container. Controlling the operating equipment to perform a delivery operation at the delivery location includes: Obtain the attitude parameters corresponding to each of the active components, as well as the external dimensions of the specified loading container; Generate a virtual object that matches the stated dimensions; Based on the posture parameters corresponding to each active component and the virtual object, the motion trajectory of the robotic arm moving to the second position is calculated; Based on the motion trajectory, the robotic arm is controlled to move to the delivery location and perform the delivery operation.

9. The method according to claim 1, characterized in that, The operation trajectory information also includes an initial pickup position. After controlling the operation device to perform the pickup operation at the first position, the method further includes: Update the initial pickup position with the first position; After determining the delivery location based on the second location and the relative positional relationship, and controlling the operating equipment to perform the delivery operation based on the delivery location, the method further includes: Control the working equipment to move to the initial pickup position.

10. The method according to claim 1, characterized in that, Before calculating the second position based on the target point cloud data and the preset point cloud model corresponding to the specified space, the method further includes: Collect spatial point cloud data corresponding to the specified space; The spatial point cloud data is filtered to obtain spatial point cloud data to be clustered; Based on Euclidean distance, the spatial point cloud data to be clustered is clustered to obtain the preset point cloud model.

11. An information control device, characterized in that, The device includes: The acquisition module is used for: In response to the trajectory recording operation, the initial pickup position is obtained; The control equipment performs a pickup operation at the initial pickup position; In response to a matching trigger operation, acquire the specified point cloud data; If the preset point cloud model corresponding to the specified space and the specified point cloud data are successfully matched, the position of the specified space relative to the working device and the attitude of the working device are recorded to obtain the working pose; wherein, the specified space is the space for loading the picked-up object; Based on the work pose, calculate the center point corresponding to the specified space, and control the work equipment to move to the center point; In response to the confirmation operation, the position of the center point relative to the working equipment is recorded to obtain the relative positional relationship, thereby acquiring the working trajectory information of the working equipment; The determination module is used to determine the target point in response to a selection operation performed on the scene image; The first control module is used to control the working equipment to perform a picking operation at a first position, where the first position is the three-dimensional coordinate position corresponding to the target point. The calculation module is used to calculate the second position based on the target point cloud data and the preset point cloud model corresponding to the specified space. The target point cloud data is the point cloud data obtained by the working equipment when it is in the working pose. The second control module is used to determine the delivery location based on the second position and the relative positional relationship, and to control the operating equipment to perform a delivery operation based on the delivery location, so as to deliver the picked-up item into the designated space.

12. An electronic device, characterized in that, The method includes a processor and a memory, the memory storing multiple instructions; the processor loads instructions from the memory to perform the steps of the information control method as described in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to execute the steps of the information control method according to any one of claims 1 to 10.

Images