Article determination method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses an article determination method and device, electronic equipment and a storage medium, and the method comprises the steps: determining the current article information in a cabin of a distribution robot in response to an article selection request triggered by a target user; if the cabin door state of the distribution robot is switched from the closed state to the open state, performing trajectory tracking on the currently selected article of the target user, and determining a first selected article based on a trajectory tracking result; if the cabin door state of the distribution robot is switched from the open state to the closed state, performing radio frequency identification on articles in a cabin body of the distribution robot to determine information of remaining articles in the cabin body of the distribution robot, and determining a second selected article based on the current article information and the information of the remaining articles; and determining a target article currently selected by the target user based on the first selected article and the second selected article. Through the technical scheme of the embodiment of the invention, the object selected by the target user can be accurately determined, object omission is avoided, and the user experience is improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to computer technology, and more particularly to an object determination method, device, electronic device and storage medium. Background Art

[0002] With the development of society and the advancement of information technology, more and more places in life use delivery robots to provide services and improve service efficiency. For example, in hotels, delivery robots can be used to solve the problem that hotel staff cannot deliver meals in time at night, thereby improving meal delivery efficiency. In another example, delivery robots can be used in Internet cafes to automatically sell items.

[0003] At present, delivery robots with automatic vending functions usually use a single sensor to determine the items selected by the user from the delivery robot, and send payment information to the user based on the selected items. However, the method of using gravity sensors for gravity sensing to determine the user's selection of items may not accurately determine the user's selection of items due to abnormal placement of items (such as items leaning on the baffle inside the delivery robot cabin), and send wrong payment information to the user. The method of using infrared sensors for position sensing to determine the user's selection of items may not accurately determine the user's selection of items due to excessive occlusion between items, and send wrong payment information to the user. It can be seen that this method of using a single sensor to determine the user's selection of items has the problem of inaccurate determination of the user's selection of items. Summary of the invention

[0004] The embodiments of the present invention provide an item determination method, device, electronic device and storage medium to accurately determine the items selected by a target user, avoid missing items, and improve the user experience.

[0005] In a first aspect, an embodiment of the present invention provides an item determination method, comprising:

[0006] In response to an item selection request triggered by a target user, determine current item information in a delivery robot cabin;

[0007] If the door state of the delivery robot is switched from a closed state to an open state, a trajectory of the currently selected item of the target user is tracked, and a first selected item of the target user is determined based on the trajectory tracking result;

[0008] If the door state of the delivery robot is switched from an open state to a closed state, radio frequency identification is performed on the items in the cabin of the delivery robot to determine the remaining item information in the cabin of the delivery robot, and the second selected item of the target user is determined based on the current item information and the remaining item information;

[0009] Based on the first selected item and the second selected item, a target item currently selected by the target user is determined.

[0010] In a second aspect, an embodiment of the present invention provides an object determination device, including:

[0011] A current item information determination module, used to determine the current item information in the delivery robot cabin in response to an item selection request triggered by a target user;

[0012] a first selected item determination module, configured to track the current selected item of the target user if the hatch state of the delivery robot is switched from a closed state to an open state, and determine the first selected item of the target user based on the track tracking result;

[0013] A second selected item determination module is used for, if the door state of the delivery robot is switched from an open state to a closed state, performing radio frequency identification on the items in the cabin of the delivery robot to determine the remaining item information in the cabin of the delivery robot, and determining the second selected item of the target user based on the current item information and the remaining item information;

[0014] The target item determination module is used to determine the target item currently selected by the target user based on the first selected item and the second selected item.

[0015] In a third aspect, an embodiment of the present invention provides an electronic device, the electronic device comprising:

[0016] one or more processors;

[0017] A memory for storing one or more programs;

[0018] When the one or more programs are executed by the one or more processors, the one or more processors implement the item determination method provided by any embodiment of the present invention.

[0019] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements an item determination method as provided in any embodiment of the present invention.

[0020] The technical solution of the embodiment of the present invention determines the current item information in the cabin of the delivery robot before the target user opens the cabin door in response to the item selection request triggered by the target user; if the cabin door state of the delivery robot is switched from the closed state to the open state, indicating that the target user opens the cabin door and selects the required item, the trajectory of the current selected item of the target user is tracked, and the first selected item of the target user is determined based on the trajectory tracking result, so as to track the trajectory of the current selected item and realize positive determination of the item selected by the target user; if the cabin door state of the delivery robot is switched from the open state to the closed state, indicating that the target user ends the item selection and closes the cabin door, the items in the cabin of the delivery robot are radio frequency identified to determine the remaining item information in the cabin of the delivery robot, and the second selected item of the target user is determined based on the current item information and the remaining item information, so as to reversely determine the item selected by the target user for the item information corresponding to each other before and after the cabin door is opened, and then accurately determine the target item selected by the target user based on the first selected item selected by the target user determined positively and the second selected item selected by the target user determined reversely, so as to avoid missing items and improve the user experience.

[0021] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 is a flow chart of an item determination method provided in Embodiment 1 of the present invention;

[0024] Figure 2 is a flow chart of an item determination method provided by Embodiment 2 of the present invention;

[0025] Figure 3 This is a flowchart of hardware acceleration involved in the second embodiment of the present invention;

[0026] Figure 4 is a structural diagram of an automatic vending system involved in Embodiment 2 of the present invention;

[0027] Figure 5 is a structural schematic diagram of a detection system involved in Embodiment 2 of the present invention;

[0028] Figure 6 is a structural schematic diagram of a delivery robot involved in Embodiment 2 of the present invention;

[0029] Figure 7 is a schematic diagram of the structure of an object determination device provided in Embodiment 3 of the present invention;

[0030] Figure 8 It is a schematic diagram of the structure of an electronic device for implementing the object determination method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0032] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0033] Embodiment 1

[0034] Figure 1 A flowchart of an item determination method is provided for the first embodiment of the present invention. This embodiment can be applied to the case of determining the item selected by the target user in the delivery robot cabin. The method can be executed by an item determination device, which can be implemented in the form of hardware and / or software, and can be configured in an electronic device. Figure 1 As shown, the method includes:

[0035] S110. In response to an item selection request triggered by a target user, determine current item information in a delivery robot cabin.

[0036] Among them, the target user may refer to the user who calls the delivery robot. The target user may select the items he needs from the delivery robot cabin. The item selection request may be used to call the delivery robot to reach the vicinity of the target user. The delivery robot may refer to a movable robot that carries a variety of items to be selected. The current item information may refer to the item information corresponding to the items in the delivery robot cabin before the target user starts to select the items in the delivery robot cabin. For example, the current item information may include: the current item category and the current item quantity corresponding to the current item category. Each item in the delivery robot cabin has an identifiable item label.

[0037] Specifically, in response to an item selection request triggered by a target user, the delivery robot may determine current item information in a cabin of the delivery robot by radio frequency identification upon receiving the item selection request.

[0038] On the basis of the above technical solution, "responding to the item selection request triggered by the target user, determining the current item information in the delivery robot cabin" may include: responding to the item selection request triggered by the target user, controlling the delivery robot to move to the target location where the target user is located; if the delivery robot moves to the target location, performing radio frequency identification in the delivery robot cabin to determine the current item information in the delivery robot cabin.

[0039] The item selection request may also include a target location set by the target user. The target location may be, but is not limited to, the location of the target user and the item selection location set by the target user.

[0040] Exemplarily, the target user places an order on a specific software, triggering an item selection request, and sends the item selection request to the cloud server, so that the cloud server determines the delivery robot closest to the target location based on the target location in the item selection request and the current location of all delivery robots, and sends an item selection request to the delivery robot. The delivery robot receives the item selection request sent by the cloud server and autonomously navigates to the target location specified by the target user. If the delivery robot moves to the target location, radio frequency identification is performed in the delivery robot cabin to determine the current item information in the delivery robot cabin, and the target user is notified by phone to accurately select items. Among them, determining that the closest delivery robot moves to the target location can quickly determine the delivery robot that responds to the item selection request and improve the efficiency of the delivery robot reaching the target location. The current item information is determined when the target location is reached, avoiding multiple determinations of the current item information before the target user starts to select items, and avoiding the determination of the current item information during the movement of the delivery robot, thereby improving the accuracy and efficiency of the determination of the current item information.

[0041] S120: If the door state of the delivery robot is switched from a closed state to an open state, a trajectory of the target user's currently selected item is tracked, and a first selected item of the target user is determined based on the trajectory tracking result.

[0042] Among them, the closed state may refer to a state in which the items in the delivery robot cabin cannot be selected. The open state may refer to a state in which the items in the delivery robot cabin can be selected. The currently selected item may refer to the item grabbed by the target user during the item selection process. The currently selected item may include the item finally selected by the target user. For example, the currently selected item may refer to the item that the target user has grabbed from the delivery robot cabin but has not yet taken out of the delivery robot cabin. The trajectory tracking result may refer to the result of trajectory tracking of the currently selected item. For example, the trajectory tracking result may include an item selection result, an item return result, and an item position adjustment result. The item selection result may indicate that the target user has selected the grabbed item. The item return result may indicate that the target user has not selected the grabbed item. The item position adjustment result may indicate that the target user has not selected the grabbed item. For example, the item position adjustment result is an item grabbing operation performed by the target user in order to grab the obscured item. The first selected item may refer to the currently selected item selected by the target user as determined positively.

[0043] Specifically, if the door state of the delivery robot switches from a closed state to an open state, indicating that the target user opens the door and selects the required item, a relevant filtering algorithm can be used to track the target user's currently selected item to generate a tracking trajectory of the currently selected item, and the tracking trajectory is judged to determine the trajectory tracking result of the currently selected item, and the first selected item of the target user is determined based on the trajectory tracking result. If the trajectory tracking result is an item selection result, the currently selected item is determined as the first selected item. If the trajectory tracking result is an item return result or an item position adjustment result, the currently selected item cannot be determined as the first selected item.

[0044] S130. If the door state of the delivery robot is switched from an open state to a closed state, radio frequency identification is performed on the items in the cabin of the delivery robot to determine the remaining item information in the cabin of the delivery robot, and the second selected item of the target user is determined based on the current item information and the remaining item information.

[0045] The remaining item information may refer to the information of the remaining items in the delivery robot cabin when the target user completes the item selection. The remaining item information may include: the remaining item category and the number of remaining items corresponding to the remaining item category. The second selected item may refer to the current selected item selected by the target user determined in reverse.

[0046] Specifically, if the door state of the delivery robot switches from an open state to a closed state, indicating that the target user has finished selecting items and closed the door, radio frequency identification is performed on the item tags in the delivery robot's cabin to determine the remaining item information in the delivery robot's cabin, and the difference item information between the current item information and the remaining item information is determined, and the second selected item of the target user is determined based on the difference item information.

[0047] On the basis of the above technical solution, "performing radio frequency identification on items in the delivery robot cabin to determine the remaining item information in the delivery robot cabin" may include: reading item labels on alternative items in the delivery robot cabin based on a preset identification frequency within a preset time period, and determining multiple alternative item information read at each identification moment; if the number of reads corresponding to the current alternative item information is greater than or equal to a preset reading number threshold, then determining the remaining item information in the delivery robot cabin based on the current alternative item information.

[0048] The preset reading number threshold may refer to a preset minimum number of times an item is read. The preset reading number threshold corresponding to each item may be the same or different. The item tag may uniquely identify the item. Items of the same category may also be distinguished by the item tag.

[0049] Specifically, when the target user finishes selecting items and closes the cabin door, the remaining item information can be determined. The item labels on the candidate items in the delivery robot cabin are read at a fixed frequency of 10Hz within a preset time period to determine the multiple candidate item information read at each identification moment; if the number of reads corresponding to the current candidate item information is less than the preset reading number threshold, it indicates that misidentification or misreading occurs, and the current candidate item does not exist. If the number of reads corresponding to the current candidate item information is greater than or equal to the preset reading number threshold, it indicates that the current candidate item exists in the delivery robot cabin, and the remaining item information in the delivery robot cabin is determined based on the current candidate item information, thereby avoiding misidentification or misreading of items in the delivery robot cabin under external interference and other circumstances, and improving the accuracy of determining the remaining item information.

[0050] S140: Determine the target item currently selected by the target user based on the first selected item and the second selected item.

[0051] The target item may refer to the item selected by the target user in the item selection request. For example, the target item may be the item that the target user needs to complete payment.

[0052] Specifically, the target item currently selected by the target user is determined based on deduplication and merging of the first selected item and the second selected item. If the first selected item and the second selected item are exactly the same, the first selected item or the second selected item is determined as the target item currently selected by the target user. If there is a difference between the first selected item and the second selected item, the entire item acquisition process of the target user is sent to manual review, and the target item currently selected by the target user is determined based on the manual review result.

[0053] On the basis of the above technical solution, "determining the target item currently selected by the target user based on the first selected item and the second selected item" may include: if there is a difference between the first selected item and the second selected item, then determining the same selected item and the different selected item based on the first selected item and the second selected item; if the difference quantity corresponding to each different selected item is less than or equal to the preset difference quantity threshold, then determining the same selected item as the target item currently selected by the target user; if the difference quantity corresponding to the different selected items is greater than the preset difference quantity threshold, then determining the secondary review information based on the different selected items greater than the preset difference quantity threshold, and determining the target item currently selected by the target user in response to the secondary review result triggered by the customer service, the same selected item and the different selected item.

[0054] Among them, the same selected items may refer to the same items between the first selected items and the second selected items. The different selected items may refer to the different items between the first selected items and the second selected items. The preset difference quantity threshold may refer to the maximum acceptable difference quantity of items. The preset difference quantity threshold corresponding to each item may be the same or different. The secondary review information may refer to the difference item information that needs to be manually reviewed. The secondary review result may refer to the review result of the customer service on the secondary review information.

[0055] Specifically, if there is no difference between the first selected item and the second selected item, the first selected item or the second selected item is determined as the target item currently selected by the target user. If there is a difference between the first selected item and the second selected item, the same selected item and the difference selected item are determined based on the first selected item and the second selected item. On the basis of determining the difference selected items, the comparison result is determined based on the difference quantity corresponding to each difference selected item and the preset difference quantity threshold corresponding to each difference selected item, and based on the comparison result, it is determined whether manual review is required for the difference selected items. If the difference quantity corresponding to each difference selected item is less than or equal to the preset difference quantity threshold, the same selected item is determined as the target item currently selected by the target user; if the difference quantity corresponding to the difference selected item is greater than the preset difference quantity threshold, the secondary review information is determined based on the difference selected item greater than the preset difference quantity threshold, and the target item currently selected by the target user is determined in response to the secondary review result triggered by the customer service, the same selected item and the difference selected item, so as to further improve the accuracy of the target item determination.

[0056] The technical solution of the embodiment of the present invention determines the current item information in the cabin of the delivery robot before the target user opens the cabin door in response to the item selection request triggered by the target user; if the cabin door state of the delivery robot is switched from a closed state to an open state, indicating that the target user opens the cabin door and selects the required item, the trajectory of the current selected item of the target user is tracked, and the first selected item of the target user is determined based on the trajectory tracking result, so as to track the trajectory of the current selected item and realize positive determination of the item selected by the target user; if the cabin door state of the delivery robot is switched from an open state to a closed state, indicating that the target user ends the item selection and closes the cabin door, the items in the cabin of the delivery robot are radio frequency identified to determine the remaining item information in the cabin of the delivery robot, and the second selected item of the target user is determined based on the current item information and the remaining item information, so as to reversely determine the item selected by the target user for the item information corresponding to each other before and after the cabin door is opened, and then accurately determine the target item selected by the target user based on the first selected item selected by the target user determined positively and the second selected item selected by the target user determined reversely, so as to avoid missing items and improve the user experience.

[0057] Embodiment 2

[0058] Figure 2 This is a flowchart of a method for determining an item provided in the second embodiment of the present invention. Based on the above embodiments, this embodiment describes in detail the process of determining the first selected item. The explanations of the terms that are the same or corresponding to the above embodiments are not repeated here. Figure 2 As shown, the method includes:

[0059] S210: In response to an item selection request triggered by a target user, determine current item information in a delivery robot cabin.

[0060] S220. If the door state of the delivery robot is switched from a closed state to an open state, the first object image in the cabin of the delivery robot is obtained, and based on the first object image and the target object recognition model, the currently selected object category, the currently selected object position and the currently selected object quantity corresponding to the currently selected object of the target user are determined.

[0061] The target object recognition model is obtained by pre-training the YOLOV8 model for object recognition. The first object image may refer to an image containing all objects in the delivery robot cabin.

[0062] Exemplarily, if the door state of the delivery robot is switched from a closed state to an open state, the first object image in the cabin of the delivery robot can be acquired at a fixed frequency of 20 Hz, and the trained target object recognition model can be used to identify the object in the first object image to determine the currently selected object category, the currently selected object position and the currently selected object quantity corresponding to the currently selected object of the target user.

[0063] It should be noted that real-time recognition of objects based on images places high demands on the computing performance of the hardware. The embodiments of the present invention can make full use of hardware acceleration technology based on the preprocessing platform to improve the timeliness and accuracy of object recognition, while greatly reducing the hardware cost. Figure 3 Describes the hardware acceleration process. Figure 3 , the image hardware acceleration unit is used to preprocess the collected image, and the preprocessed image is sent to the target object recognition model after quantization and inference for object recognition. Among them, the trained target object recognition model can be quantized first, and then the neural network operation unit is used to perform model inference on the quantized model. Specifically, the image preprocessing can be performed by the image hardware acceleration unit (RGA) built into the preprocessing platform to perform color gamut conversion, scaling, rotation and cropping operations on the input image, replacing the software calculation that originally needs to be performed on the CPU, thereby improving the preprocessing efficiency. Model quantization can quantize the trained target object recognition model through hybrid quantization technology, which on the one hand compresses the model volume and reduces the space required for model storage; on the other hand, it improves the model calculation speed while the accuracy is basically not affected. Model inference can be performed on the above quantized model through the neural network operation unit (NPU) built into the preprocessing platform. Compared with CPU inference, the speed is increased by dozens of times, ensuring the timeliness of object recognition.

[0064] S230: If it is detected for the first time that the position of the currently selected object reaches the preset tracking start position, the trajectory of the currently selected object is tracked to determine the trajectory tracking result corresponding to the currently selected object.

[0065] The preset tracking start position may refer to a preset track tracking start position. For example, the preset tracking start position may be, but is not limited to, the position of the item when it leaves the shelf and is in the air. The advantage of such a setting is that the track tracking can be performed only on the item grabbed by the target user, thereby narrowing the scope of the track tracking items and further improving the efficiency of determining the first selected item.

[0066] S240: Determine the first selected item of the target user based on the trajectory tracking result, the category of the currently selected item, and the number of the currently selected items.

[0067] For example, using the above example, if the track tracking result is an item selection result, the category and quantity of the target user's first selected item are determined using the currently selected item category and the currently selected item quantity, thereby determining the target user's first selected item. If the track tracking result is an item return result or an item position adjustment result, the currently selected item category and the currently selected item quantity cannot be used to determine the target user's first selected item.

[0068] Based on the above technical solution, "determining the first selected item of the target user based on the trajectory tracking results, the category of the currently selected items and the number of currently selected items" may include: if the trajectory tracking result is that the currently selected item moves from inside the delivery robot cabin to outside the delivery robot cabin, and the end of the tracking trajectory is outside the delivery robot cabin, then determining the first selected item of the target user based on the category of the currently selected item and the number of currently selected items.

[0069] Among them, the item selection result may include that the currently selected item moves from the delivery robot cabin to the delivery robot cabin, and the end of the tracking track is outside the delivery robot cabin. Specifically, if the track tracking result is that the currently selected item moves from the delivery robot cabin to the delivery robot cabin, and the end of the tracking track is outside the delivery robot cabin, indicating that the target user has selected the currently selected item and has not returned the selected currently selected item, then the category and quantity of the target user's first selected item are determined using the category and quantity of the currently selected item, thereby determining the target user's first selected item, and then determining whether the currently selected item is the first selected item based on the track tracking result, further improving the accuracy of determining the first selected item.

[0070] S250: If the door state of the delivery robot is switched from an open state to a closed state, radio frequency identification is performed on the items in the cabin of the delivery robot to determine the remaining item information in the cabin of the delivery robot, and the second selected item of the target user is determined based on the current item information and the remaining item information.

[0071] S260: Determine the target item currently selected by the target user based on the first selected item and the second selected item.

[0072] The technical solution of the embodiment of the present invention obtains the first item image in the delivery robot cabin, and determines the current selected item category, current selected item position and current selected item quantity corresponding to the current selected item of the target user based on the first item image and the target item recognition model; the target item recognition model is obtained by pre-training the YOLOV8 model for item recognition; thereby, the pre-trained target item recognition model is used to perform image recognition to determine the current selected item selected by the target user and various attribute information of the current selected item, such as category, position and quantity, to improve the accuracy of determining the current selected item and the current selected item information. If it is detected for the first time that the position of the current selected item reaches the preset tracking starting position, the trajectory of the current selected item is tracked to determine the trajectory tracking result corresponding to the current selected item; the first selected item of the target user is determined based on the trajectory tracking result, the category of the current selected item and the number of the current selected items, thereby using the position information to filter whether to track the current selected item, narrowing the range of items that need to be tracked, and further improving the efficiency of determining the first selected item.

[0073] On the basis of the above technical solution, the method also includes: if the trajectory tracking of the currently selected item fails, then determining the predicted item position corresponding to the currently selected item by performing position prediction based on the existing tracking trajectory corresponding to the currently selected item; obtaining a second item image corresponding to the predicted item position in the delivery robot cabin, and determining the candidate item category and the number of candidate items corresponding to the candidate item at the predicted item position based on the second item image and the target item recognition model; if the candidate item category is the same as the currently selected item category, and the number of candidate items is the same as the currently selected item number, then determining the candidate item as the currently selected item, and continuing to track the trajectory of the currently selected item.

[0074] The existing tracking track may refer to the tracking track that has been generated in the tracking process before the tracking fails. The predicted object position may refer to the position to which the currently selected object will move at the next moment. The second object image may refer to an image including the object at the predicted object position. For example, the second object image may be, but is not limited to, an image that only includes the object at the predicted object position. The candidate object may refer to the object included in the second object image.

[0075] Specifically, when tracking the trajectory of the currently selected item, object occlusion or camera occlusion may occur, resulting in failure of tracking the trajectory of the currently selected item. If the trajectory tracking of the currently selected item fails, the predicted item position corresponding to the currently selected item is determined based on the existing tracking trajectory corresponding to the currently selected item and the Kalman filter for position prediction; the second item image corresponding to the predicted item position in the delivery robot cabin is obtained, and the candidate item category and the number of candidate items corresponding to the candidate item at the predicted item position are determined based on the second item image and the target item recognition model; if the candidate item category is the same as the currently selected item category, and the number of candidate items is the same as the currently selected item number, the candidate item is determined as the currently selected item, and the trajectory tracking of the currently selected item is continued, so as to predict in advance the predicted item position where the currently selected item may appear, and perform item recognition again for the predicted item position, accurately determine the currently selected item, and continue to track the trajectory, thereby improving the anti-interference performance in the process of determining the first selected item, and further improving the accuracy of determining the target item.

[0076] Based on the above technical solution, the embodiment of the present invention also provides an automatic vending system using the item determination method. Figure 4 Describe the system. Figure 4 , the system may include a delivery robot and a cloud server. The delivery robot may include a detection system and a front cabin. The delivery robot may be a robot used for automated delivery and delivery, with functions such as autonomous navigation, obstacle avoidance and path planning, and can automatically drive and complete the delivery task of items according to the preset destination and route. The front cabin may be a small warehouse pre-installed on the delivery robot, storing specific items for retail, which can increase the delivery speed and reduce the transportation time. The detection system can execute the item determination method in the embodiment of the present invention and perform settlement based on the accurately determined target items.

[0077] Combination Figure 5 Describes the detection system. Figure 5 ,The detection system may include a sensor module, a computing module and a communication module.

[0078] Among them, the sensor module is responsible for collecting relevant data of the user's pickup process and sending the data to the computing module for processing and calculation. The sensor module mainly includes a radio frequency reader and writer module and a video detection module. The radio frequency identification module is located at the bottom of the front cabin and is connected to the computing module through a serial interface. The radio frequency reader reads the label attached to the item at a fixed frequency and distinguishes the type of item by the label data. When the item is in the front cabin, the item data can be read, and when the item is taken away, the data cannot be read. The pickup situation can be preliminarily judged by the changes in the data before and after. The video detection module is located above the front cabin and is connected to the computing module through a USB interface. The detection system uses a camera with less distortion and higher frame rate to capture images of the user's pickup process. Through a deep learning algorithm, it can identify and track the items taken by the user in real time, and determine whether they have been taken out by the movement trajectory of the items.

[0079] The computing module is responsible for receiving and processing the sensor module data, and sending the processing results to the communication module. The computing module mainly includes a storage unit, an operation unit and a peripheral circuit. The storage unit includes a hard disk and a memory. The hard disk is used to install the software operation detection system, save the detection program and the detection results, and the memory is used to temporarily store the operation data of the operation unit and the data exchanged with the peripherals. The operation unit mainly includes an operator and a controller. The operation unit is used to run the detection program on the hard disk, perform the calculation of the shipment detection, store the detection results in the hard disk and the memory, perform corresponding processing according to the detection results, and convert the processing results into corresponding control instructions and send them to the communication module. The peripheral circuit mainly includes a power supply circuit and a peripheral interface. The peripheral circuit provides power output for the storage unit and the operation unit. The peripheral interface is used to connect the sensor module and the communication module. The serial port and USB interface in the peripheral interface are connected to the sensor module.

[0080] The communication module connects to the Internet through the 4G network module, sends the detection results of this detection system to the cloud server, and receives and processes the information sent by the cloud server. The communication module mainly includes the application module and the network module. The application module is responsible for parsing and processing data, completing protocol conversion and data encapsulation, and the network module is responsible for receiving and sending data.

[0081] Combination Figure 6 Describes the delivery robot. Figure 6 , the delivery robot may include a screen, a video detection module, a front cabin, a computing module, a communication module, a cabin door, and a radio frequency identification module. The user operates the delivery robot through the screen above. Below the screen is the video detection module. The video detection module looks down at a certain angle to the front, which is convenient for capturing images of the user picking up the goods. Items can be placed in the front cabin, and there are cabin doors on the left and right sides, which are opened and closed by motors. The computing module and the communication module are integrated together and located on the back of the front cabin. At the bottom of the cabin is the radio frequency identification module, which can identify the tags of items above it and inside the cabin.

[0082] The object determination device provided in the embodiment of the present invention can execute the object determination method provided in any embodiment of the present invention, and has the corresponding beneficial effects of executing the object determination method.

[0083] The following is an embodiment of an item determination device provided in an embodiment of the present invention. The device and the item determination methods in the above embodiments belong to the same inventive concept. For details not described in detail in the embodiment of the item determination device, reference can be made to the embodiment of the above-mentioned item determination method.

[0084] Embodiment 3

[0085] Figure 7 This is a schematic diagram of the structure of an item identification device provided in Embodiment 3 of the present invention. Figure 7 As shown, the device includes: a current item information determination module 710, a first selected item determination module 720, a second selected item determination module 730 and a target item determination module 740.

[0086] Among them, the current item information determination module 710 is used to determine the current item information in the delivery robot cabin in response to the item selection request triggered by the target user; the first selected item determination module 720 is used to track the trajectory of the target user's current selected item if the door state of the delivery robot is switched from a closed state to an open state, and determine the target user's first selected item based on the trajectory tracking result; the second selected item determination module 730 is used to perform radio frequency identification on the items in the delivery robot cabin to determine the remaining item information in the delivery robot cabin if the door state of the delivery robot is switched from an open state to a closed state, and determine the target user's second selected item based on the current item information and the remaining item information; the target item determination module 740 is used to determine the target item currently selected by the target user based on the first selected item and the second selected item.

[0087] The technical solution of the embodiment of the present invention determines the current item information in the cabin of the delivery robot before the target user opens the cabin door in response to the item selection request triggered by the target user; if the cabin door state of the delivery robot is switched from a closed state to an open state, indicating that the target user opens the cabin door and selects the required item, the trajectory of the current selected item of the target user is tracked, and the first selected item of the target user is determined based on the trajectory tracking result, so as to track the trajectory of the current selected item and realize positive determination of the item selected by the target user; if the cabin door state of the delivery robot is switched from an open state to a closed state, indicating that the target user ends the item selection and closes the cabin door, the items in the cabin of the delivery robot are radio frequency identified to determine the remaining item information in the cabin of the delivery robot, and the second selected item of the target user is determined based on the current item information and the remaining item information, so as to reversely determine the item selected by the target user for the item information corresponding to each other before and after the cabin door is opened, and then accurately determine the target item selected by the target user based on the first selected item selected by the target user determined positively and the second selected item selected by the target user determined reversely, so as to avoid missing items and improve the user experience.

[0088] On the basis of the above technical solution, the current item information determination module 710 is specifically used to: respond to the item selection request triggered by the target user, control the delivery robot to move to the target location where the target user is located; if the delivery robot moves to the target location, perform radio frequency identification in the delivery robot cabin to determine the current item information in the delivery robot cabin.

[0089] Based on the above technical solution, the first selected item determination module 720 may include:

[0090] The currently selected item information determination submodule is used to obtain the first item image in the delivery robot cabin, and determine the currently selected item category, the currently selected item position and the currently selected item quantity corresponding to the currently selected item of the target user based on the first item image and the target item recognition model; the target item recognition model is obtained by pre-training the YOLOV8 model for item recognition;

[0091] A trajectory tracking result determination submodule is used to track the trajectory of the currently selected object and determine the trajectory tracking result corresponding to the currently selected object if it is detected for the first time that the position of the currently selected object reaches the preset tracking start position;

[0092] The first selected item determination submodule is used to determine the first selected item of the target user based on the trajectory tracking result, the category of the current selected item and the number of the current selected items.

[0093] Based on the above technical solution, the first selected item determination submodule is specifically used for: if the trajectory tracking result is that the currently selected item moves from inside the delivery robot cabin to outside the delivery robot cabin, and the end of the tracking trajectory is outside the delivery robot cabin, then the first selected item of the target user is determined based on the currently selected item category and the currently selected item quantity.

[0094] On the basis of the above technical solution, the device also includes:

[0095] A predicted item position determination module is used to determine the predicted item position corresponding to the currently selected item by performing position prediction based on the existing tracking trajectory corresponding to the currently selected item if the trajectory tracking of the currently selected item fails;

[0096] A candidate item information determination module is used to obtain a second item image corresponding to the predicted item position in the delivery robot cabin, and determine the candidate item category and the number of candidate items corresponding to the candidate item at the predicted item position based on the second item image and the target item recognition model;

[0097] The track tracking module is used to determine the candidate item as the currently selected item if the category of the candidate item is the same as the category of the currently selected item, and the number of the candidate items is the same as the number of the currently selected items, and continue to track the track of the currently selected item.

[0098] On the basis of the above technical solution, the second selected item determination module 730 is specifically used to: read the item labels on the alternative items in the delivery robot cabin based on a preset recognition frequency within a preset time period, and determine the multiple alternative item information read at each recognition moment; if the number of reads corresponding to the current alternative item information is greater than or equal to the preset reading number threshold, then determine the remaining item information in the delivery robot cabin based on the current alternative item information.

[0099] On the basis of the above technical solution, the target item determination module 740 is specifically used for: if there is a difference between the first selected item and the second selected item, then determining the same selected item and the different selected item based on the first selected item and the second selected item; if the difference quantity corresponding to each different selected item is less than or equal to the preset difference quantity threshold, then determining the same selected item as the target item currently selected by the target user; if the difference quantity corresponding to the different selected items is greater than the preset difference quantity threshold, then determining the secondary review information based on the different selected items greater than the preset difference quantity threshold, and determining the target item currently selected by the target user in response to the secondary review result triggered by the customer service, the same selected item and the different selected item.

[0100] The object determination device provided in the embodiment of the present invention can execute the object determination method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the object determination method.

[0101] It is worth noting that in the embodiments determined by the above-mentioned items, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0102] Embodiment 4

[0103] Figure 8 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0104] like Figure 8 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0105] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0106] The processor 11 may be a variety of general and / or dedicated processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as an item determination method.

[0107] In some embodiments, the item determination method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the item determination method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the item determination method in any other appropriate manner (e.g., by means of firmware).

[0108] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0109] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0110] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0111] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0112] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0113] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0114] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0115] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for determining an item, characterized in that: include: In response to an item selection request triggered by a target user, determine current item information in a delivery robot cabin; If the door state of the delivery robot is switched from a closed state to an open state, a trajectory of the currently selected item of the target user is tracked, and a first selected item of the target user is determined based on the trajectory tracking result; If the door state of the delivery robot is switched from an open state to a closed state, radio frequency identification is performed on the items in the cabin of the delivery robot to determine the remaining item information in the cabin of the delivery robot, and the second selected item of the target user is determined based on the current item information and the remaining item information; Based on the first selected item and the second selected item, a target item currently selected by the target user is determined.

2. The method according to claim 1, characterized in that The step of determining the current item information in the delivery robot cabin in response to the item selection request triggered by the target user includes: In response to an item selection request triggered by a target user, controlling the delivery robot to move to a target location where the target user is located; If the delivery robot moves to the target location, radio frequency identification is performed in the delivery robot cabin to determine current item information in the delivery robot cabin.

3. The method according to claim 1, characterized in that The step of tracking the current selected item of the target user and determining the first selected item of the target user based on the track tracking result includes: Acquire a first object image in the delivery robot cabin, and determine the current selected object category, current selected object position, and current selected object quantity corresponding to the current selected object of the target user based on the first object image and a target object recognition model; the target object recognition model is obtained by pre-training the YOLOV8 model for object recognition; If it is detected for the first time that the position of the currently selected object reaches the preset tracking start position, the trajectory of the currently selected object is tracked to determine the trajectory tracking result corresponding to the currently selected object; The first selected item of the target user is determined based on the trajectory tracking result, the category of the currently selected item and the number of the currently selected items.

4. The method according to claim 3, characterized in that: The determining the first selected item of the target user based on the trajectory tracking result, the category of the currently selected item, and the number of the currently selected items includes: If the trajectory tracking result is that the currently selected item moves from inside the delivery robot cabin to outside the delivery robot cabin, and the end of the tracking trajectory is outside the delivery robot cabin, the first selected item of the target user is determined based on the category of the currently selected item and the number of the currently selected items.

5. The method according to claim 3, characterized in that: The method further comprises: If the trajectory tracking of the currently selected object fails, a position prediction is performed based on the existing tracking trajectory corresponding to the currently selected object to determine the predicted object position corresponding to the currently selected object; Acquire a second object image corresponding to the predicted object position in the delivery robot cabin, and determine the candidate object category and the number of candidate objects corresponding to the candidate object at the predicted object position based on the second object image and the target object recognition model; If the category of the candidate item is the same as the category of the currently selected item, and the number of the candidate items is the same as the number of the currently selected items, the candidate item is determined as the currently selected item, and the trajectory of the currently selected item continues to be tracked.

6. The method according to claim 1, characterized in that The step of performing radio frequency identification on the items in the delivery robot cabin to determine the remaining items information in the delivery robot cabin includes: Reading item labels on candidate items in the delivery robot cabin based on a preset recognition frequency within a preset time period, and determining information of multiple candidate items read at each recognition moment; If the number of times the current candidate item information is read is greater than or equal to a preset reading number threshold, the remaining item information in the delivery robot cabin is determined based on the current candidate item information.

7. The method according to claim 1, characterized in that The determining the target item currently selected by the target user based on the first selected item and the second selected item includes: If there is a difference between the first selected item and the second selected item, determining the same selected item and the different selected items based on the first selected item and the second selected item; If the difference quantity corresponding to each different selected item is less than or equal to the preset difference quantity threshold, the same selected item is determined as the target item currently selected by the target user; If the difference quantity corresponding to the difference selected items is greater than the preset difference quantity threshold, the secondary review information is determined based on the difference selected items greater than the preset difference quantity threshold, and the target item currently selected by the target user is determined in response to the secondary review result triggered by the customer service, the same selected items and the difference selected items.

8. An object identification device, characterized in that: include: A current item information determination module, used to determine the current item information in the delivery robot cabin in response to an item selection request triggered by a target user; a first selected item determination module, configured to track the current selected item of the target user if the hatch state of the delivery robot is switched from a closed state to an open state, and determine the first selected item of the target user based on the track tracking result; A second selected item determination module is used for, if the door state of the delivery robot is switched from an open state to a closed state, performing radio frequency identification on the items in the cabin of the delivery robot to determine the remaining item information in the cabin of the delivery robot, and determining the second selected item of the target user based on the current item information and the remaining item information; The target item determination module is used to determine the target item currently selected by the target user based on the first selected item and the second selected item.

9. An electronic device, characterized in that: The electronic device comprises: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the item determination method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the object determination method according to any one of claims 1 to 7 is implemented.

Images