Robot control method and related device, robot and storage medium

Abstract

The present application discloses a robot control method and related devices, a robot and a storage medium, wherein the robot control method includes: obtaining the current captured image of the robot at the stacking location of the current goods to be picked up; performing target detection based on the current captured image to obtain several image areas of the current goods to be picked up and their confidences, and obtaining a detection threshold based on the detection analysis results of the reference captured image and the quality detection results of the current captured image; screening each image area based on the detection threshold and the confidence of the image area to obtain several target areas of the current goods to be picked up; based on the relationship between the first number of several target areas and the second number of the current goods to be picked up in the order to be picked up, at least controlling whether the robot should feedback a stock-out prompt. The above scheme can reduce the hardware cost and algorithm complexity while ensuring the accuracy of the robot's out-of-stock detection as much as possible.

Description

Technical Field

[0001] The present application relates to the field of robotics technology, and in particular to a robot control method and related devices, a robot and a storage medium. Background Art

[0002] With the rapid development of robot technology, robots have been used in more and more scenarios. For example, in the commodity pickup scenario, by deploying robots to pick up orders, efficiency can be greatly improved.

[0003] Before the robot performs the picking action for each item in the order, it is inevitable to perform out-of-stock detection first. Existing robots usually combine visible light images and depth images to perform out-of-stock detection to ensure the accuracy of out-of-stock detection as much as possible. However, this method requires additional hardware costs for depth cameras on the one hand, and the algorithm complexity is usually high on the other hand because it needs to combine two images. In view of this, how to reduce hardware costs and algorithm complexity while ensuring the accuracy of out-of-stock detection performed by robots as much as possible has become an urgent problem to be solved. Summary of the invention

[0004] The main technical problem solved by the present application is to provide a robot control method and related devices, a robot and a storage medium, which can reduce the hardware cost and algorithm complexity while ensuring the accuracy of the robot's out-of-stock detection as much as possible.

[0005] In order to solve the above-mentioned technical problems, the first aspect of the present application provides a robot control method, including: obtaining a current captured image of the robot at the stacking location of the current goods to be picked up; performing target detection based on the current captured image to obtain several image areas of the current goods to be picked up and their confidence levels, and obtaining a detection threshold based on the detection analysis results of the reference captured image and the quality detection results of the current captured image; wherein the reference captured image is a historical captured image of the robot performing target detection, the detection analysis results represent the accuracy of the target detection results, and the quality detection results represent the degree of image quality; based on the detection threshold and the confidence level of the image area, each image area is screened separately to obtain several target areas of the current goods to be picked up; based on the relationship between the first quantity of the several target areas and the second quantity of the current goods to be picked up in the order to be picked up, at least control whether the robot should feedback an out-of-stock prompt.

[0006] In order to solve the above technical problems, the second aspect of the present application provides a robot control device, including: an acquisition module, a verification module, a screening module, and a control module. The acquisition module is used to acquire the current captured image of the robot at the stacking location of the current goods to be picked up; the verification module is used to perform target detection based on the current captured image, obtain several image areas of the current goods to be picked up and their confidence levels, and obtain a detection threshold based on the detection and analysis results of the reference captured image and the quality detection results of the current captured image; wherein the reference captured image is a historical captured image of the robot performing target detection, the detection and analysis results represent the accuracy of the target detection results, and the quality detection results represent the degree of image quality; the screening module is used to screen each image area based on the detection threshold and the confidence level of the image area, and obtain several target areas of the current goods to be picked up; the control module is used to control at least whether the robot feeds back an out-of-stock prompt based on the relationship between the first quantity of the several target areas and the second quantity of the current goods to be picked up in the order to be picked up.

[0007] In order to solve the above technical problems, the third aspect of the present application provides an electronic device, which at least includes a memory and a processor coupled to each other, the memory at least stores program instructions, and the processor is used to execute the program instructions to implement the robot control method in the above first aspect.

[0008] In order to solve the above-mentioned technical problems, the fourth aspect of the present application provides a robot, which at least includes a driving device, a navigation device, a camera device and the electronic device in the above-mentioned third aspect, the camera device is used to take images, and the driving device drives the overall movement of the robot under the guidance of the navigation device.

[0009] In order to solve the above technical problems, the fifth aspect of the present application provides a computer-readable storage medium storing program instructions that can be executed by a processor, and the program instructions are used to implement the robot control method of the first aspect.

[0010] In the above scheme, the current captured image of the robot at the stacking location of the current commodity to be picked up is obtained, and then target detection is performed based on the current captured image to obtain several image areas of the current commodity to be picked up and their confidences. The detection threshold is obtained based on the detection analysis result of the reference captured image and the quality detection result of the current captured image, and the reference captured image is a historical captured image of the robot performing target detection. The detection analysis result represents the accuracy of the target detection result, and the quality detection result represents the degree of image quality. Therefore, based on the detection threshold and the confidence of the image area, each image area is screened respectively to obtain several target areas of the current commodity to be picked up, and then based on the relationship between the first number of the several target areas and the second number of the current commodity to be picked up in the order to be picked up, at least whether the robot feeds back a stock-out prompt is controlled. Therefore, on the one hand, since only a single image needs to be detected without combining multiple images, the hardware cost and algorithm complexity can be reduced. On the other hand, by adjusting the detection threshold in combination with the detection analysis result of the reference captured image and the quality detection result of the current captured image, the detection threshold can be adaptively changed with the actual situation, and the image area is screened accordingly, so as to ensure the accuracy of the robot performing stock-out detection as much as possible. Therefore, the hardware cost and algorithm complexity can be reduced while ensuring the accuracy of the robot's out-of-stock detection as much as possible. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 It is a flow chart of an embodiment of the robot control method of the present application;

[0012] Figure 2 It is a process schematic diagram of an embodiment of the robot control method of the present application;

[0013] Figure 3 It is a schematic diagram of the framework of an embodiment of the robot control device of the present application;

[0014] Figure 4 It is a schematic diagram of the framework of an embodiment of the electronic device of the present application;

[0015] Figure 5 It is a schematic diagram of the framework of an embodiment of the robot of the present application;

[0016] Figure 6 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0017] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0018] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0019] The terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the fragment " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, "many" in this article means two or more than two.

[0020] See also Figure 1 , Figure 1 This is a flow chart of an embodiment of the robot control method of the present application. Specifically, it may include the following steps:

[0021] Step S11: obtaining the current image captured by the robot at the location where the goods to be picked up are stacked.

[0022] In one implementation scenario, the current product to be picked up is the product that the robot currently needs to pick up in the order to be picked up. It should be noted that the order to be picked up may contain several products to be picked up, and the robot can select each product to be picked up in turn according to a specific strategy as the current product to be picked up. In other words, when the robot takes a product to be picked up as the current product to be picked up and completes the picking operation or detects that the product to be picked up is out of stock, the next product to be picked up can be used as the new current product to be picked up.

[0023] In an implementation scenario, please refer to Figure 2 , Figure 2 1 is a schematic diagram of a process of an embodiment of the robot control method of the present application. Figure 2 As shown, the order to be picked up can be obtained by placing an order through the ordering system by a smart terminal (such as a smart phone, a tablet computer, etc.). It should be noted that the ordering system can be presented in the form of a web page, so as to realize a cross-platform system and can adapt to a variety of different situations. In addition, the web page of the ordering system can display the types of various commodities and buttons such as ordering, and then the commodities and quantities can be selected on the web page according to actual needs. After confirmation, click the order button to submit the order to be picked up. Alternatively, the ordering system can also be implemented through an APP. In addition, the robot can also directly receive the order to be picked up through voice input and other methods. In other words, it is also possible to place an order directly to the robot through voice. For example, the robot can be awakened by a voice wake-up word first, and after the robot responds, the name and quantity of the required commodity can be said to form an order to be picked up.

[0024] In an implementation scenario, please continue to refer to Figure 2After the robot obtains the order to be picked up, it can parse the order to be picked up, such as obtaining the order data in JSON format. On this basis, a planned path for executing the pickup operation for the order to be picked up can be formed according to the stacking location of each commodity and each commodity contained in the order to be picked up (for example, path planning can be performed based on A*, Dijkstra algorithm, etc.), and the planned path can include the corresponding points of the stacking location of each commodity in the order to be picked up, so as to move along the planned path, stop at the corresponding point of the stacking location of the commodity in the order to be picked up, take pictures, and perform subsequent related steps. Specifically, when the robot is controlled to reach the stacking location, it can stop and take pictures (for example, an RGB camera can be used to capture color images, or a fill light can be further configured to obtain the clearest possible image even under low light conditions), and then the image can be detected and processed (for example, an image processing unit GPU can be used for processing at the hardware level, and a neural network model can be used for processing at the software level).

[0025] It should be noted that the smart terminal can be integrated with a front-end order module (such as for placing orders in the aforementioned web pages, APPs, etc.), and the robot can be integrated with a voice interaction module (such as for the aforementioned voice interaction), an order processing module (such as for parsing orders to be picked up), a navigation module (such as for forming the aforementioned planned path and controlling the movement of the robot), etc., and different modules can communicate through websocket. For example, the front-end order module can interact with the order processing module through websocket, and the voice interaction module can also interact with the order processing module through websocket. Examples are not given one by one here. For the sake of convenience of description, take the order to be picked up as an example, which contains three items, namely, item A to be picked up, item B to be picked up, and item C to be picked up, and the planned path is to first move to the stacking place of item A to be picked up, then move to the stacking place of item B to be picked up, and finally move to the stacking place of item C to be picked up. This example will also be used in conjunction with specific examples later.

[0026] Step S12: Target detection is performed based on the current captured image to obtain several image regions of the current commodity to be picked up and their confidence levels, and a detection threshold is obtained based on the detection analysis results of the reference captured image and the quality detection results of the current captured image.

[0027] In one implementation scenario, a neural network model such as YOLO can be used to perform target detection on the current captured image to obtain several image areas of the current commodity to be picked up in the current captured image and their confidence levels. Still taking the aforementioned planned path as an example, when traveling to commodity A to be picked up, commodity A to be picked up is the current commodity to be picked up, so target detection can be performed on the current captured image at its stacking location to obtain several image areas of commodity A to be picked up in the current captured image and their confidence levels. It should be noted that the greater the confidence level of an image area, the higher the possibility that the commodity to be picked up is contained in the image area, and conversely, the smaller the confidence level of an image area, the lower the possibility that the commodity to be picked up is contained in the image area.

[0028] In a specific implementation scenario, the neural network model can be trained using a training data set before it is formally applied. For example, the training data set can include multiple pictures of products at different angles and under different lighting conditions, and marked with the real area and real category (or real name) of the products. In addition, the training data set can also include pictures of products that have been taken from real products and have been enhanced (e.g., rotated, scaled, flipped, etc.) to maximize the robustness of the neural network model after the neural network model is trained accordingly.

[0029] In a specific implementation scenario, as mentioned above, the neural network model can adopt target detection models such as YOLO. In addition, the neural network model can also be improved on the basis of target detection models such as YOLO. For example, the backbone network in the target detection model can adopt the latest deep learning architecture, introduce residual connections to directly pass the input to the subsequent layers through jump connections to solve the gradient vanishing problem in the deep network, and use the attention mechanism to enhance the network's attention to important features. Of course, the above examples are just a few examples of improvements to the target detection model, and other possible improvements will not be given one by one here.

[0030] In one implementation scenario, using residual connection as an example, feature extraction can be performed based on the current captured image to obtain a first feature map, and then at least a convolution operation can be performed based on the first feature map to obtain a second feature map, so that prediction can be performed based on the global features of the second feature map to obtain a first weight representation, and prediction can be performed based on the local features of the second feature map to obtain a second weight representation, and then based on the first weight representation and the second weight representation, a fused weight representation can be obtained, and then the first feature map and the second feature map can be fused based on the fused weight representation to obtain a fused feature map, and prediction can be performed based on the fused feature map to obtain several image regions and their confidences. The above method, by performing weight prediction on the global features and local features of the second feature map respectively, to fuse the global weight representation and the local weight representation, can not only pay attention to the global features but also the local features in the process of fusion features, which helps to improve the detection accuracy of various scales of goods in the current captured image.

[0031] In a specific implementation scenario, as a possible example, after obtaining the fused feature map, regional prediction may not be performed temporarily, but the fused feature map may be used as a new first feature map, and the above step of performing at least a convolution operation based on the first feature map is returned to obtain the second feature map and iterated to fuse the local features and the global features multiple times. It should be noted that the number of iterations can be set according to the actual application needs. For example, when the fusion demand is large, the number of iterations can be set appropriately larger, or, when the fusion demand is relatively loose, the number of iterations can be set appropriately smaller. The specific value of the number of iterations is not limited here.

[0032] In a specific implementation scenario, after obtaining the first feature map, a convolution may be performed on the first feature map, and then an activation function (such as a linear rectifier activation function, etc.) may be used to activate the feature map after the convolution, and finally a second convolution may be performed on the activated feature map to obtain the second feature map. Of course, the above example is only a possible example of obtaining the second feature map, and does not limit the possible ways of performing other operations on the first feature map to obtain the second feature map, and no examples are given here one by one.

[0033] In a specific implementation scenario, after obtaining the second feature map, global average pooling can be performed in the channel dimension based on the second feature map to obtain a first weight representation. Exemplarily, the number of channels of the second feature map is N, then after global average pooling is performed in the channel dimension, the obtained first weight representation can be expressed as a data size of N*1, that is, it contains the weight value of each channel. Of course, the above example is only a possible example of global weights, and does not limit other possible ways to obtain global weights, so no examples are given here one by one.

[0034] In a specific implementation scenario, after obtaining the second feature map, the second feature map can be processed based on the sliding window to obtain a local feature map, and prediction can be performed based on the local feature map to obtain a second weight representation. It should be noted that the embodiment of the present disclosure does not limit the size, stride, etc. of the sliding window. In addition, after obtaining the local features, the local features can be input into the fully connected layer for prediction to obtain a second weight representation, and the second weight representation has the same size as the first weight representation (such as the aforementioned N*1). Of course, the above example is only a possible example of local weights, and does not limit other possible ways of obtaining local weights, and no examples are given here one by one.

[0035] In a specific implementation scenario, after obtaining the first weight representation and the second weight representation, the first weight representation and the second weight representation may be fused by summing, weighting, averaging, etc. to obtain a fused weight representation. Exemplarily, still taking the first weight representation and the second weight representation as an example of a size of N*1, the element values ​​at the same position in the first weight representation and the second weight representation may be added, weighted, or averaged to obtain the element values ​​at the corresponding position in the fused weight representation of size N*1.

[0036] In a specific implementation scenario, after obtaining the fusion weight representation, the fusion weight representation can be used to perform a weighted operation on the second feature map to obtain a weighted feature map. Specifically, the sub-feature maps of the corresponding channels in the second feature map can be weighted using the values ​​of each element in the fusion weight representation to obtain a weighted feature map. For example, the sub-feature map of the i-th channel in the second feature map can be weighted using the i-th element value in the fusion weight representation to obtain the sub-feature map of the i-th channel in the weighted feature map. On this basis, the weighted feature map can be added to the first feature map, such as adding the element values ​​at the same position on the sub-feature map of the same channel in the weighted feature map to the first feature map, and finally obtaining a fused feature map. It should be noted that the above example is only a possible example of fusing the first feature map and the second feature map using the fusion weight representation, and does not limit other possible ways of fusing the first feature map and the second feature map using the fusion weight representation, and other possible ways are not given examples one by one here.

[0037] In a specific implementation scenario, after experiencing at least one communication and fusion between the features at different levels, not only can a higher level of abstract representation (these high-level features contain feature information such as the shape and texture of the goods to be picked up) be gradually formed as the network depth increases, but also a richer feature representation (i.e., fused feature map) can be obtained through continuous fusion. On this basis, the fused feature map can be predicted. Exemplarily, still taking YOLO as an example, the fused feature map can be input into detection heads of different scales, so that each detection head can make predictions at different scales to predict the image areas and their confidences of the goods to be picked up of different sizes in the current captured image. Of course, in actual application, other different prediction methods can also be used, and examples are not given one by one here.

[0038] In the disclosed embodiment, the reference captured image is a historical captured image of the robot performing target detection, the detection and analysis result represents the accuracy of the target detection result, and the quality detection result represents the degree of image quality. Exemplarily, as mentioned above, the robot moves along the planned path, and can perform tasks such as out-of-stock and picking up goods in turn for each item to be picked up in the pickup order. The reference captured image can be an image captured by the robot at the stacking location of the previous item to be picked up. Of course, when the robot performs target detection for the first time, since there are no historical captured images before this, the detection threshold can be obtained based only on the quality detection result of the current captured image. For details, please refer to the following related description, which will not be repeated here.

[0039] In one implementation scenario, statistics can be performed on the image areas obtained when performing target detection based on historical captured images to obtain the false detection rate, so as to determine the accuracy represented by the detection and analysis results of the reference captured images. For example, after the robot performs target detection on the historical captured images, it can obtain several image areas detected for a certain commodity to be picked up in the historical captured images, and the robot can perform operations such as picking up or prompting out of stock based on this (for example, if the total number of several image areas is not less than the required quantity of the commodity to be picked up in the order to be picked up, then the goods are picked up, otherwise, it is prompted that the commodity is out of stock). When picking up the goods, the robot can use the mechanical arm to perform picking up in the detected image area to determine whether there is a false detection in the image area, so that the false detection rate can be calculated, or when prompting out of stock, the background can perform more accurate target detection (such as manual review, etc.) based on the historical captured images, so that the false detection rate can be calculated; or, after the robot performs target detection on the historical captured images, it can obtain several image areas detected for a certain commodity to be picked up in the historical captured images, and the robot can directly transmit this detection result back to the background, so that the background can calculate the false detection rate. The above examples are only several possible examples of statistical false positive rate, and do not limit other possible ways of statistical false positive rate, and other possible ways will not be given examples one by one here. On this basis, the accuracy represented by the detection and analysis results of the reference captured image can be determined according to the false positive rate. For example, the higher the false positive rate, the lower the accuracy represented by the detection and analysis results of the reference captured image. On the contrary, the lower the false positive rate, the higher the accuracy represented by the detection and analysis results of the reference captured image.

[0040] In an implementation scenario, the quality detection result can be specifically represented by the indicator values ​​of several quality indicators, and the several quality indicators include at least one of light intensity, clarity, and contrast. As a possible example, the light intensity can be evaluated by calculating the average brightness value of the current captured image. For example, the light intensity can be represented as:

[0041] …… (1)

[0042] In the above formula (1), W and H represent the width and height of the current captured image respectively. Represents the brightness value of pixel (i, j). As a possible example, clarity can be evaluated by calculating the gradient amplitude of the current captured image. For example, clarity can be expressed as:

[0043] …… (2)

[0044] In the above formula (2), Represents the gradient of pixel (i, j). As a possible example, the contrast can be evaluated by the local contrast of the current captured image. For example, the contrast can be expressed as:

[0045] …… (3)

[0046] In the above formula (3), Represents the average brightness value of the current captured image. Of course, the above examples are only the calculation methods of the three quality indicators of light intensity, clarity, and contrast, and are not limited to other quality indicators. No more examples are given here. As a possible example, a small neural network (such as a network layer that can include convolutional layers, pooling layers, etc.) can be used in combination with the above formula to evaluate image quality. For example, image data of different lighting conditions and clarity can be collected in advance as a training data set, and each sample image in the training data set can be annotated with annotation data such as real light intensity, real clarity, and real contrast, and the training data set is used to train the above small neural network to constrain the predicted light intensity output by the small neural network model to be as close to the real light intensity as possible, the predicted clarity to be as close to the real clarity as possible, and the predicted contrast to be as close to the real contrast as possible. On this basis, the current captured image can be directly input into the above small neural network model, so that the index values ​​of the quality indicators such as light intensity, clarity, and contrast of the current captured image can be obtained.

[0047] In one implementation scenario, as described above, when the robot performs target detection for the first time, since there is no historical captured image before, the detection threshold can be obtained based only on the quality detection result of the current captured image. Exemplarily, the degree represented by the quality detection result of the current captured image and the detection threshold can be positively correlated. In other words, the higher the image quality of the current captured image, the larger the detection threshold can be; or, the lower the image quality of the current captured image, the lower the detection threshold can be, so as to improve the detection sensitivity.

[0048] In another implementation scenario, different from the aforementioned method, when the robot is not performing target detection for the first time, the detection threshold can be obtained by combining the detection analysis results of the reference captured image and the quality detection results of the current captured image. Specifically, the initial threshold can be obtained based on the quality detection results of the current captured image, and then the initial threshold can be adjusted based on the detection analysis results of the reference captured image to obtain the detection threshold. In the above method, the initial threshold is first determined based on the quality detection results, and then the initial threshold is adjusted based on the detection analysis results. Therefore, on the one hand, the detection threshold can be determined by taking into account both the quality detection results and the detection analysis results, and on the other hand, the quality detection results can be given priority while taking into account both, and the detection analysis results can be considered secondarily.

[0049] In a specific implementation scenario, the initial threshold may be positively correlated with the image quality. For example, the higher the image quality, the larger the initial threshold, and vice versa, the lower the image quality, the smaller the initial threshold. As a possible example, the image quality can be determined by the index value of the light intensity, and the initial threshold can be expressed as:

[0050] Threshold(light intensity) = f(light intensity) …… (4)

[0051] The above formula (4) indicates that there is a functional relationship f between the light intensity and the initial threshold. Specifically, the functional relationship f indicates that the lower the light intensity, the smaller the initial threshold, and vice versa, the greater the light intensity, the greater the initial threshold. As a possible example, the degree of image quality can be determined by the index value of clarity, and the initial threshold can be expressed as:

[0052] Threshold(clarity) = g(clarity) …… (5)

[0053] The above formula (5) indicates that there is a functional relationship g between the clarity and the initial threshold. Specifically, the functional relationship g indicates that the lower the clarity, the smaller the initial threshold, and vice versa, the higher the clarity, the larger the initial threshold. As a possible example, the degree of image quality can be determined by the index value of contrast, and the initial threshold can be expressed as:

[0054] Threshold(contrast) = h(contrast) …… (6)

[0055] The above formula (6) indicates that there is a functional relationship h between contrast and the initial threshold. Specifically, the functional relationship h indicates that the lower the contrast, the smaller the initial threshold, and vice versa, the higher the contrast, the larger the initial threshold. As a possible example, the image quality can be determined by the index values ​​of the three quality indicators of illumination intensity, clarity, and contrast. Then the final initial threshold can be obtained by weighting the initial thresholds determined by the above three quality indicators, which can be expressed as:

[0056] Comprehensive threshold = w1×threshold (light intensity) + w2×threshold (clarity) + w3×threshold (contrast) …… (7)

[0057] In the above formula (7), w1, w2, and w3 represent the weight values ​​of the three quality indicators of light intensity, clarity, and contrast, respectively.

[0058] In a specific implementation scenario, after obtaining the initial threshold, the initial threshold can be adjusted according to the detection analysis result of the reference captured image to obtain the detection threshold. Specifically, the adjustment direction (e.g., increase or decrease) and the adjustment range (i.e., how much to increase or decrease) of the initial threshold can be determined based on the accuracy represented by the detection analysis result, and the adjustment direction is determined by the magnitude relationship between the accuracy and the reference, and the adjustment range is determined by the degree difference between the accuracy and the reference. Exemplarily, as mentioned above, the accuracy can be determined by the false detection rate, and the reference can also be determined by a preset false detection rate (e.g., 15%, 20%, etc.). If the false detection rate is larger than the preset false detection rate, the adjustment direction can be determined to be reduced, and vice versa, if the false detection rate is smaller than the preset false detection rate, the adjustment direction can be determined to be increased. In addition, if the difference between the false detection rate and the preset false detection rate is larger, it can be determined that the adjustment range is larger, and vice versa, if the difference between the false detection rate and the preset false detection rate is smaller, it can be determined that the adjustment range is smaller. On this basis, the initial threshold can be adjusted based on the adjustment direction and adjustment amplitude to obtain the detection threshold.

[0059] In another implementation scenario, as another possible implementation mode, the detection and analysis result of the reference captured image and the quality detection result of the current captured image can be expressed as a functional relationship f:

[0060] …… (8)

[0061] That is to say, the detection analysis result of the reference captured image and the quality detection result of the current captured image can be directly input into the functional relationship f to obtain the detection threshold. It should be noted that the above functional relationship f can be obtained by data fitting or represented by a small neural network, which is not limited here.

[0062] Step S13: Screening each image area based on the detection threshold and the confidence of the image area to obtain several target areas of the current commodity to be picked up.

[0063] Specifically, after obtaining the detection threshold, the image area where the current commodity to be picked up is detected in the current captured image can be screened accordingly. For example, for each image area, it can be determined whether the confidence of the image area is not lower than the detection threshold. If so, the image area can be retained, otherwise the image area can be eliminated. The image areas that are finally retained can be used as several target areas of the current commodity to be picked up in the current captured image.

[0064] Step S14: Based on the relationship between the first quantity of the plurality of target areas and the second quantity of the current commodities to be picked up in the order to be picked up, at least control the robot to feedback an out-of-stock prompt.

[0065] In one implementation scenario, if the first quantity is not less than the second quantity, it can be determined that the current goods to be picked up are not out of stock, and the robot may not feedback the out-of-stock prompt. In addition, the robot can also retrieve the second quantity of the current goods to be picked up at the stacking place of the current goods to be picked up. As a possible example, when the order to be picked up contains multiple goods to be picked up, the robot can continue to move to the stacking place of the next goods to be picked up, return to execute the aforementioned step S11 and perform loop iterations until all the goods to be picked up in the order to be picked up have completed the above steps. Alternatively, if the first quantity is less than the second quantity, it can be determined that the current goods to be picked up are out of stock, and the robot can feedback the out-of-stock prompt. As a possible example, the robot can also temporarily retrieve the first quantity of the current goods to be picked up at the stacking place of the current goods to be picked up, and the remaining current retrieved goods that need to be retrieved can continue to retrieve after the current goods to be picked up are in sufficient stock; as another possible example, the robot can also temporarily not retrieve the current goods to be picked up, and after the current goods to be picked up are in sufficient stock, retrieve the second quantity of the current goods to be picked up at the stacking place of the current goods to be picked up.

[0066] In one implementation scenario, considering related scenarios where orders frequently exist during peak shopping seasons, in response to receiving a new order to be picked up, it can be first determined whether there is a robot whose current state is idle. It should be noted that if the robot is in an idle state, it means that the robot is not performing a pickup task, that is, it can be scheduled to perform a pickup task for a new order to be picked up. As a possible example, if it is determined to exist, the robot whose current state is idle can be directly scheduled to perform a pickup task for the new order to be picked up (at this time, the current state of the robot is switched from idle to working), that is, it can return to execute the above step S11 and iterate in a loop until all the goods to be picked up in the new order to be picked up have been executed. As another possible example, when it is determined that there is no such order, the picking tasks of at least part of the goods to be picked up in the new order can be assigned to at least part of the robots based on the orders to be picked up that each robot is responsible for and the current location, and wait until there is a robot that is currently in an idle state, and then the picking tasks that have not been assigned in the new order to be picked up are assigned to the robot that is currently in an idle state, so that the robot that is currently in an idle state can perform the step of obtaining the current image taken by the robot at the stacking location of the current goods to be picked up. In the above method, when a new order to be picked up is received, according to the absence of a robot that is currently in an idle state, the picking tasks of part of the goods to be picked up in the new order to be picked up are selected to be assigned to some robots that are performing picking tasks, and the picking tasks that cannot be assigned are assigned when a robot that is currently in an idle state appears, which can minimize the impact on robots in a working state and improve the execution efficiency of picking tasks for new orders to be picked up.

[0067] In a specific implementation scenario, when allocating picking tasks for each item to be picked up in a new order to be picked up, each item to be picked up in the new order to be picked up can be selected separately as the first target item to be assigned the picking task. In response to the existence of a first robot among the robots, the picking task of the first target item can be preferentially allocated to the first robot, and the first robot satisfies: the first target item exists in the order to be picked up for which it is responsible and the stacking location of the first target item in the original planned path does not lag behind the current location of the first robot. Taking a new order to be picked up as an example, which includes goods A to be picked up, goods B to be picked up, and goods C to be picked up, when selecting goods A to be picked up as the first target goods to be assigned to the picking task, if there is a robot among the robots in the working state, and the goods A to be picked up also exists in the order to be picked up that it is responsible for, and the stacking location of goods A to be picked up in the original planned path of the robot does not lag behind the current location of the robot, that is, the current location of the robot is exactly the stacking location of goods A to be picked up, or the robot will reach the stacking location of goods A to be picked up in the future if it continues to move from its current location, then the robot can be used as the first robot, and the picking task of goods A to be picked up in the new order to be picked up is assigned to the first robot. In this way, when the first robot performs the picking task of goods A to be picked up in its original order to be picked up, it can first retrieve a certain number of goods A to be picked up in its original order to be picked up, and then retrieve a certain number of goods A to be picked up in the new order to be picked up. Of course, if out-of-stock situations occur during this process, you can refer to the above-mentioned related descriptions for handling, and will not go into details here.

[0068] In a specific implementation scenario, in response to the absence of the first robot but the presence of the second robot, the task of picking up the first target product can be secondarily assigned to the second robot, and the second robot meets the following conditions: the first target product does not exist in the order to be picked up that it is responsible for, the original planned path passes through the stacking location of the first target product, and the stacking location of the first target product in the original planned path does not lag behind the current location of the second robot. Still taking the new order to be picked up as an example, which includes the goods to be picked up A, goods to be picked up B and goods to be picked up C, when selecting the goods to be picked up B as the first target goods to be assigned the picking task, if the aforementioned first robot does not exist among the robots in the working state, but there is a robot, although the goods to be picked up B does not exist in the order to be picked up that it is responsible for, its original planned path will pass through the stacking place of the goods to be picked up B, and the stacking place of the goods to be picked up B in the original planned path does not lag behind the current position of the robot, that is, the current position of the robot is exactly the stacking place of the goods to be picked up B, or if the robot continues to move at the current position, it will reach the stacking place of the goods to be picked up B in the future, then the robot can be used as the second robot, and the picking task of the goods to be picked up B in the new order to be picked up is assigned to the second robot. In this way, when the second robot passes the stacking place of the goods to be picked up B, it can retrieve a certain number of the goods to be picked up B in the new order to be picked up. Of course, if a situation such as out of stock occurs during this process, it can be handled with reference to the above-mentioned related description, which will not be repeated here.

[0069] In a specific implementation scenario, in response to the absence of the first robot and the second robot, the task of picking up the first target commodity can be reserved. Still taking the new order to be picked up as an example, including the commodity A to be picked up, the commodity B to be picked up, and the commodity C to be picked up, when the commodity C to be picked up is selected as the first target commodity to be assigned a picking up task, if the first robot and the second robot do not exist among the robots in the working state, the task of picking up the commodity C to be picked up is reserved, and a robot that is currently in an idle state will be left to perform the task of picking up the commodity C to be picked up.

[0070] In another implementation scenario, different from the aforementioned implementation, considering the relevant scenarios where there are frequent orders such as the peak shopping season, in response to receiving a new order to be picked up, each of the goods to be picked up in the new order to be picked up can be selected as the second target goods to be assigned the picking up task, and for the second target goods, it is determined whether there is a target robot among the robots that are performing the picking up task, and the target robot satisfies: the second target goods exist in the order to be picked up that it is responsible for and the stacking location of the second target goods in the original planned path does not lag behind the current location of the target robot. On this basis, if it is judged to exist, the picking up task of the second target goods can be assigned to the target robot, and if it is judged not to exist, the picking up task of the second target goods can be retained. It should be noted that the specific meaning of assigning the picking up task of the second target goods to the target robot can refer to the aforementioned description of assigning the picking up task of the first target goods to the first robot (or the second robot), which will not be repeated here. In the above method, after receiving a new order to be picked up, the picking task for each item to be picked up in the new order to be picked up is assigned to the target robot that is performing the task. This can improve the execution efficiency of the picking task for the new order to be picked up while minimizing the impact on the robot in working state.

[0071] In a specific implementation scenario, still taking the new order to be picked up containing goods A, goods B and goods C as an example, when selecting goods A as the second target goods to be assigned to the picking task, if there is a robot among the robots that are performing the picking task, the order to be picked up for which it is responsible already has goods A, and the stacking location of goods A in the original planned path does not lag behind its current location. In other words, the current location of the robot is exactly the stacking location of goods A, or the robot will reach the stacking location of goods A in the future if it continues to move along the original planned path at its current location. In this case, the robot can be used as the target robot, and the picking task of goods A can be assigned to the target robot.

[0072] In a specific implementation scenario, still taking the new order to be picked up containing goods A, goods B and goods C as an example, when selecting goods B as the second target goods to be assigned to the picking task, if there is no target robot that meets the above requirements among the robots that are performing the picking task, that is, either there is no goods B in the order to be picked up that it is responsible for, or there is goods B but the stacking location of goods B in the original planned path lags behind its current location, then the picking task of goods B can be retained. It should be noted that when retaining the picking task of the second target goods, it can be assigned to the robot that is currently in the idle state. For details, please refer to the relevant description of retaining the first target goods, which will not be repeated here.

[0073] It should be noted that the three methods described in the embodiments of the present disclosure: only completely allocating the task of picking up goods in a new order to be picked up to the robot that is currently in an idle state, selectively allocating the task of picking up goods in a new order to be picked up to the robot in a working state when there is no robot that is currently in an idle state, and preferentially allocating the task of picking up goods in a new order to be picked up to the robot in a working state regardless of whether there is a robot that is currently in an idle state, are just a few possible methods in actual application, and the specific method of allocating picking tasks is not limited here.

[0074] In the above scheme, the current captured image of the robot at the stacking location of the current commodity to be picked up is obtained, and then target detection is performed based on the current captured image to obtain several image areas of the current commodity to be picked up and their confidences. The detection threshold is obtained based on the detection analysis result of the reference captured image and the quality detection result of the current captured image, and the reference captured image is a historical captured image of the robot performing target detection. The detection analysis result represents the accuracy of the target detection result, and the quality detection result represents the degree of image quality. Therefore, based on the detection threshold and the confidence of the image area, each image area is screened respectively to obtain several target areas of the current commodity to be picked up, and then based on the relationship between the first number of the several target areas and the second number of the current commodity to be picked up in the order to be picked up, at least whether the robot feeds back a stock-out prompt is controlled. Therefore, on the one hand, since only a single image needs to be detected without combining multiple images, the hardware cost and algorithm complexity can be reduced. On the other hand, by adjusting the detection threshold in combination with the detection analysis result of the reference captured image and the quality detection result of the current captured image, the detection threshold can be adaptively changed with the actual situation, and the image area is screened accordingly, so as to ensure the accuracy of the robot performing stock-out detection as much as possible. Therefore, the hardware cost and algorithm complexity can be reduced while ensuring the accuracy of the robot's out-of-stock detection as much as possible.

[0075] See also Figure 3 , Figure 3 It is a schematic diagram of a framework of an embodiment of the robot control device of the present application. The robot control device 30 includes: an acquisition module 31, a verification module 32, a screening module 33, and a control module 34. The acquisition module 31 is used to acquire the current captured image of the robot at the stacking location of the current goods to be picked up; the verification module 32 is used to perform target detection based on the current captured image, obtain several image areas of the current goods to be picked up and their confidences, and obtain the detection threshold based on the detection analysis results of the reference captured image and the quality detection results of the current captured image; wherein the reference captured image is a historical captured image of the robot performing target detection, the detection analysis results represent the accuracy of the target detection results, and the quality detection results represent the degree of image quality; the screening module 33 is used to screen each image area based on the detection threshold and the confidence of the image area, and obtain several target areas of the current goods to be picked up; the control module 34 is used to control whether the robot at least feeds back a shortage prompt based on the relationship between the first quantity of the several target areas and the second quantity of the current goods to be picked up in the order to be picked up.

[0076] In the above scheme, the robot control device 30 obtains the current captured image of the robot at the stacking location of the current goods to be picked up, and then performs target detection based on the current captured image to obtain several image areas of the current goods to be picked up and their confidences, and obtains the detection threshold based on the detection analysis result of the reference captured image and the quality detection result of the current captured image, and the reference captured image is a historical captured image of the robot performing target detection, the detection analysis result represents the accuracy of the target detection result, and the quality detection result represents the degree of image quality, so that each image area is screened based on the detection threshold and the confidence of the image area to obtain several target areas of the current goods to be picked up, and then based on the relationship between the first number of the several target areas and the second number of the current goods to be picked up in the order to be picked up, at least control whether the robot feeds back a stock-out prompt. Therefore, on the one hand, since only a single image needs to be detected without combining multiple images, the hardware cost and algorithm complexity can be reduced, and on the other hand, by adjusting the detection threshold in combination with the detection analysis result of the reference captured image and the quality detection result of the current captured image, the detection threshold can be adaptively changed with the actual situation, and the image area is screened accordingly, so as to ensure the accuracy of the robot's out-of-stock detection as much as possible. Therefore, the hardware cost and algorithm complexity can be reduced while ensuring the accuracy of the robot's out-of-stock detection as much as possible.

[0077] In some disclosed embodiments, the verification module 32 includes an initial threshold determination submodule, which is used to obtain an initial threshold based on the quality detection result of the current captured image; the verification module 32 includes an initial threshold adjustment submodule, which is used to adjust the initial threshold based on the detection analysis result of the reference captured image to obtain a detection threshold.

[0078] In some disclosed embodiments, the initial threshold is positively correlated to the high or low degree.

[0079] In some disclosed embodiments, the initial threshold adjustment submodule includes a direction amplitude determination unit, which is used to determine the adjustment direction and adjustment amplitude of the initial threshold based on the degree of accuracy; wherein the adjustment direction is determined by the magnitude relationship between the degree of accuracy and the reference degree, and the adjustment amplitude is determined by the degree difference between the degree of accuracy and the reference degree; the initial threshold adjustment submodule includes an initial threshold adjustment unit, which is used to adjust the initial threshold based on the adjustment direction and adjustment amplitude to obtain a detection threshold.

[0080] In some disclosed embodiments, the verification module 32 includes a feature extraction submodule for performing feature extraction based on the current captured image to obtain a first feature map; the verification module 32 includes a convolution operation submodule for performing at least a convolution operation based on the first feature map to obtain a second feature map; the verification module 32 includes a weight prediction submodule for performing prediction based on the global features of the second feature map to obtain a first weight representation, and performing prediction based on the local features of the second feature map to obtain a second weight representation, and obtaining a fused weight representation based on the first weight representation and the second weight representation; the verification module 32 includes a feature fusion submodule for fusing the first feature map and the second feature map based on the fused weight representation to obtain a fused feature map; the verification module 32 includes a region prediction submodule for performing prediction based on the fused feature map to obtain a number of image regions and their confidence levels.

[0081] In some disclosed embodiments, the weight prediction submodule includes a global prediction unit, which is used to perform global average pooling in the channel dimension based on the second feature map to obtain a first weight representation; the weight prediction submodule includes a local prediction unit, which is used to process the second feature map based on a sliding window to obtain a local feature map, and perform prediction based on the local feature map to obtain a second weight representation.

[0082] In some disclosed embodiments, the robot control device 30 includes a state judgment module for judging whether there is a robot whose current state is idle in response to receiving a new order to be picked up; the robot control device 30 includes an initial allocation module for allocating the picking tasks of at least part of the goods to be picked up in the new order to be picked up to at least part of the robots based on the orders to be picked up that each robot is responsible for and the current location, if it is judged that there is no such order; the robot control device 30 includes a re-allocation module for waiting until there is a robot whose current state is idle, and then allocating the picking tasks that have not been allocated in the new order to be picked up to the robot whose current state is idle, so that the robot whose current state is idle can execute the step of obtaining the current image taken by the robot at the stacking location of the current goods to be picked up.

[0083] In some disclosed embodiments, the initial allocation module includes a commodity selection submodule, which is used to select each commodity to be picked up in a new order to be picked up, respectively, as the first target commodity to be assigned for the picking task to be assigned; the initial allocation module includes a first allocation submodule, which is used to preferentially allocate the picking task of the first target commodity to the first robot in response to the existence of the first robot among the robots; wherein the first robot satisfies: the first target commodity exists in the order to be picked up for which it is responsible and the stacking location of the first target commodity in the original planned path does not lag behind the current location of the first robot; the initial allocation module includes a second allocation submodule, which is used to secondarily allocate the picking task of the first target commodity to the second robot in response to the existence of the second robot but the absence of the first robot; wherein the second robot satisfies: the first target commodity does not exist in the order to be picked up for which it is responsible, the original planned path passes through the stacking location of the first target commodity and the stacking location of the first target commodity in the original planned path does not lag behind the current location of the second robot; the initial allocation module includes a task reservation submodule, which is used to reserve the picking task of the first target commodity in response to the absence of the first robot and the second robot.

[0084] In some disclosed embodiments, the robot control device 30 includes a selection and judgment module for, in response to receiving a new order to be picked up, selecting each of the goods to be picked up in the new order to be picked up as the second target goods to be assigned a picking task, and judging whether there is a target robot among the robots that are performing the picking task for the second target goods; wherein the target robot satisfies: the second target goods exist in the order to be picked up that it is responsible for and the stacking location of the second target goods in the original planned path does not lag behind the current location of the target robot; the robot control device 30 includes a task assignment module for assigning the picking task of the second target goods to the target robot when it is judged that the second target goods exist; the robot control device 30 includes a task retention module for retaining the picking task of the second target goods when it is judged that the second target goods do not exist.

[0085] In some disclosed embodiments, the quality detection result is represented by indicator values ​​of several quality indicators, and the several quality indicators include at least one of light intensity, clarity, and contrast.

[0086] See also Figure 4 , Figure 4 4 is a schematic diagram of a framework of an embodiment of an electronic device of the present application. The electronic device 40 includes at least a memory 41 and a processor 42 coupled to each other, the memory 41 at least stores program instructions, and the processor 42 is used to execute the program instructions to implement the steps in any of the above robot control method embodiments. For details, please refer to the aforementioned disclosed embodiments, which will not be repeated here. As a possible example, the electronic device 40 may include but is not limited to an industrial computer, etc., and the specific type of the electronic device 40 is not limited here.

[0087] Specifically, the processor 42 is used to control itself and the memory 41 to implement the steps in any of the above robot control method embodiments. The processor 42 can also be called a CPU (Central Processing Unit). The processor 42 may be an integrated circuit chip with signal processing capabilities. The processor 42 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 42 can be implemented by an integrated circuit chip.

[0088] In the above scheme, the electronic device 40 obtains the current captured image of the robot at the stacking location of the current goods to be picked up, and then performs target detection based on the current captured image to obtain several image areas of the current goods to be picked up and their confidences, and obtains the detection threshold based on the detection analysis result of the reference captured image and the quality detection result of the current captured image, and the reference captured image is a historical captured image of the robot performing target detection, the detection analysis result represents the accuracy of the target detection result, and the quality detection result represents the degree of image quality, so that each image area is screened based on the detection threshold and the confidence of the image area to obtain several target areas of the current goods to be picked up, and then based on the relationship between the first number of the several target areas and the second number of the current goods to be picked up in the order to be picked up, at least control whether the robot feeds back a stock-out prompt. Therefore, on the one hand, since only a single image needs to be detected without combining multiple images, the hardware cost and algorithm complexity can be reduced, and on the other hand, by adjusting the detection threshold in combination with the detection analysis result of the reference captured image and the quality detection result of the current captured image, the detection threshold can be adaptively changed with the actual situation, and the image area is screened accordingly, so as to ensure the accuracy of the robot's out-of-stock detection as much as possible. Therefore, the hardware cost and algorithm complexity can be reduced while ensuring the accuracy of the robot's out-of-stock detection as much as possible.

[0089] See also Figure 5 , Figure 5: is a schematic diagram of a framework of an embodiment of the robot of the present application. The robot 50 at least includes a driving device 51, a navigation device 52, a camera device 53 and the electronic device 40 in the above-mentioned electronic device embodiment. The camera device 53 is used to capture images. The driving device 51 drives the robot 50 to move as a whole under the guidance of the navigation device 52. As a possible example, the driving device 51, the navigation device 52, and the camera device 53 can be coupled to the electronic device 40 respectively.

[0090] In the above scheme, the robot 50 at least includes a driving device 51, a navigation device 52, a camera device 53 and the electronic device 40 in the above electronic device embodiment. The camera device 53 is used to capture images. The driving device 51 drives the robot 50 to move as a whole under the guidance of the navigation device 52. Therefore, on the one hand, since only a single image needs to be detected without combining multiple images, the hardware cost and algorithm complexity can be reduced. On the other hand, by adjusting the detection threshold by combining the detection analysis results of the reference captured image and the quality detection results of the current captured image, the detection threshold can be adaptively changed according to the actual situation, and the image area can be screened accordingly, so as to ensure the accuracy of the robot's out-of-stock detection as much as possible. Therefore, the hardware cost and algorithm complexity can be reduced under the premise of ensuring the accuracy of the robot's out-of-stock detection as much as possible.

[0091] See also Figure 6 , Figure 6 The computer-readable storage medium 60 stores program instructions 61 that can be executed by a processor, and the program instructions 61 are used to implement the steps in any of the above robot control method embodiments.

[0092] In the above scheme, the computer-readable storage medium 60 obtains the current captured image of the robot at the stacking location of the current commodity to be picked up, and then performs target detection based on the current captured image to obtain several image areas of the current commodity to be picked up and their confidences, and obtains the detection threshold based on the detection analysis result of the reference captured image and the quality detection result of the current captured image, and the reference captured image is a historical captured image of the robot performing target detection, the detection analysis result represents the accuracy of the target detection result, and the quality detection result represents the degree of image quality, so that each image area is screened based on the detection threshold and the confidence of the image area to obtain several target areas of the current commodity to be picked up, and then based on the relationship between the first number of the several target areas and the second number of the current commodity to be picked up in the order to be picked up, at least control whether the robot feeds back a stock-out prompt. Therefore, on the one hand, since only a single image needs to be detected without combining multiple images, the hardware cost and algorithm complexity can be reduced, and on the other hand, by adjusting the detection threshold in combination with the detection analysis result of the reference captured image and the quality detection result of the current captured image, the detection threshold can be adaptively changed with the actual situation, and the image area is screened accordingly, so as to ensure the accuracy of the robot performing stock-out detection as much as possible. Therefore, the hardware cost and algorithm complexity can be reduced while ensuring the accuracy of the robot's out-of-stock detection as much as possible.

[0093] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0094] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0095] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0096] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0097] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0098] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0099] If the technical solution of this application involves personal information, the product using the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using the technical solution of this application has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to the collection of his or her personal information; or on the device that processes personal information, the personal information processing rules are notified by obvious signs / information, and the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

Claims

1. A robot control method, characterized in that: include: Obtain the current image taken by the robot at the stacking location of the current product to be picked up; Performing target detection based on the current captured image to obtain several image regions of the current commodity to be picked up and their confidences, and obtaining a detection threshold based on the detection analysis result of the reference captured image and the quality detection result of the current captured image; wherein the reference captured image is a historical captured image of the robot performing the target detection, the detection analysis result represents the accuracy of the target detection result, and the quality detection result represents the quality of the image; Based on the detection threshold and the confidence of the image area, each of the image areas is screened to obtain several target areas of the current commodity to be picked up; Based on the relationship between the first quantity of the plurality of target areas and the second quantity of the current commodities to be picked up in the order to be picked up, at least controlling the robot to feedback a shortage prompt; The detection threshold is obtained based on the detection analysis result of the reference captured image and the quality detection result of the current captured image, including: Obtaining an initial threshold based on a quality detection result of the current captured image; The initial threshold is adjusted based on the detection analysis result of the reference captured image to obtain the detection threshold.

2. The method according to claim 1, characterized in that The initial threshold is positively correlated with the high and low levels.

3. The method according to claim 1, characterized in that The step of adjusting the initial threshold based on the detection analysis result of the reference captured image to obtain the detection threshold comprises: Based on the accuracy, determining the adjustment direction and adjustment range of the initial threshold; wherein the adjustment direction is determined by the magnitude relationship between the accuracy and the reference level, and the adjustment range is determined by the degree difference between the accuracy and the reference level; The initial threshold is adjusted based on the adjustment direction and the adjustment amplitude to obtain the detection threshold.

4. The method according to claim 1, characterized in that: The target detection is performed based on the current captured image to obtain several image regions of the current commodity to be picked up and their confidence levels, including: Perform feature extraction based on the current captured image to obtain a first feature map; Perform at least a convolution operation based on the first feature map to obtain a second feature map; Predicting based on the global features of the second feature map to obtain a first weighted representation, predicting based on the local features of the second feature map to obtain a second weighted representation, and obtaining a fused weighted representation based on the first weighted representation and the second weighted representation; Based on the fusion weight representation, the first feature map and the second feature map are fused to obtain a fused feature map; Prediction is performed based on the fused feature map to obtain the several image regions and their confidence levels.

5. The method according to claim 4, characterized in that The predicting based on the global features of the second feature map to obtain the first weighted representation includes: performing global average pooling in the channel dimension based on the second feature map to obtain the first weighted representation; And / or, predicting based on the local features of the second feature map to obtain the second weighted representation includes: processing the second feature map based on a sliding window to obtain a local feature map, and predicting based on the local feature map to obtain the second weighted representation.

6. The method according to claim 1, characterized in that The method further comprises: In response to receiving a new order to be picked up, determining whether there is a robot that is currently in an idle state; If it is determined that the goods do not exist, based on the orders to be picked up that the robots are responsible for and their current locations, the task of picking up at least part of the goods to be picked up in the new order to be picked up is assigned to at least part of the robots; Wait until there is a robot whose current state is the idle state, and then assign the unassigned picking tasks in the new order to be picked up to the robot whose current state is the idle state, so that the robot whose current state is the idle state can execute the step of obtaining the current image taken by the robot at the stacking location of the current goods to be picked up.

7. The method according to claim 6, characterized in that The step of assigning the task of picking up at least part of the commodities to be picked up in the new order to be picked up to at least part of the robots based on the order to be picked up that each of the robots is responsible for and the current location of the order, comprises: Selecting each of the commodities to be picked up in the new order to be picked up as the first target commodity to be assigned the picking up task; In response to the presence of a first robot among the robots, the task of picking up the first target product is preferentially assigned to the first robot; wherein the first robot satisfies: the first target product exists in the order to be picked up that it is responsible for and the stacking location of the first target product in the original planned path does not lag behind the current location of the first robot; In response to the absence of the first robot but the presence of the second robot, the task of picking up the first target commodity is secondarily assigned to the second robot; wherein the second robot meets the following conditions: the first target commodity does not exist in the order to be picked up, the original planned path passes through the stacking location of the first target commodity, and the stacking location of the first target commodity in the original planned path does not lag behind the current location of the second robot; In response to the absence of the first robot and the second robot, the task of picking up the first target commodity is reserved.

8. The method according to claim 1, characterized in that The method further comprises: In response to receiving a new order to be picked up, each of the items to be picked up in the new order to be picked up is selected as a second target item to be assigned a picking task, and for the second target item, it is determined whether there is a target robot among the robots that are performing the picking task; wherein the target robot satisfies: the second target item exists in the order to be picked up that it is responsible for and the stacking location of the second target item in the original planned path does not lag behind the current location of the target robot; If it is determined that the second target commodity exists, assigning a task of picking up the second target commodity to the target robot; If it is determined that the second target commodity does not exist, the task of picking up the second target commodity is retained.

9. The method according to any one of claims 1 to 8, characterized in that: The quality detection result is represented by indicator values ​​of several quality indicators, and the several quality indicators include at least one of light intensity, clarity, and contrast.

10. A robot control device, characterized in that: include: An acquisition module is used to acquire the image currently captured by the robot at the stacking location of the current product to be picked up; A verification module, configured to perform target detection based on the current captured image, obtain a number of image regions of the current commodity to be picked up and their confidence levels, and obtain a detection threshold based on a detection analysis result of a reference captured image and a quality detection result of the current captured image; wherein the reference captured image is a historical captured image of the robot performing the target detection, the detection analysis result represents the accuracy of the target detection result, and the quality detection result represents the quality of the image; A screening module, used for screening each of the image areas based on the detection threshold and the confidence of the image area, to obtain several target areas of the current commodity to be picked up; A control module is used to control whether the robot at least feeds back a stock-out prompt based on a relationship between the first quantity of the plurality of target areas and the second quantity of the current goods to be picked up in the order to be picked up; wherein the detection threshold is obtained based on the detection analysis result of the reference captured image and the quality detection result of the current captured image, including: Obtaining an initial threshold based on a quality detection result of the current captured image; The initial threshold is adjusted based on the detection analysis result of the reference captured image to obtain the detection threshold.

11. An electronic device, characterized in that: The robot control method comprises at least a memory and a processor coupled to each other, wherein the memory at least stores program instructions, and the processor is used to execute the program instructions to implement the robot control method according to any one of claims 1 to 9.

12. A robot, characterized in that: The robot comprises at least a driving device, a navigation device, a camera device and the electronic device as claimed in claim 11, wherein the camera device is used to take images, and the driving device drives the robot to move as a whole under the guidance of the navigation device.

13. A computer-readable storage medium, characterized in that: Program instructions that can be executed by a processor are stored, and the program instructions are used to implement the robot control method according to any one of claims 1 to 9.

Images