Method for picking goods by using intelligent wearable equipment, and equipment and device for picking goods in warehouse

Abstract

The invention relates to a method for picking goods by using intelligent wearable equipment, and equipment and a device for picking goods in a warehouse. The method comprises the steps that environment data collected by an intelligent wearable device is acquired, the environment data comprises voice data associated with order sorting goods of a deliveryman and image data associated with order sorting goods of the deliveryman, the voice data is analyzed to determine a voice instruction associated with a sorting order of the deliveryman, and the voice instruction is sent to the intelligent wearable device; identifying the image data to determine goods information associated with the sorting order of the deliveryman, and according to at least one of the voice instruction, the goods information and the behavior stage of the deliveryman for picking goods in the target warehouse at present, obtaining guide content information related to the sorting order and output to the deliveryman wearing the intelligent wearable device; and outputting guide content information related to the sorting order through a voice playing mode of the intelligent wearable device or an augmented reality projection mode. By adopting the method, the order picking efficiency of the deliveryman can be improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of warehouse picking management, in particular, to a picking method using a smart wearable device, a device and an apparatus for warehouse picking. BACKGROUND

[0002] Warehouse picking management, due to the large number of goods in the warehouse and the scattered distribution, if relying on the memory or simple marking of the workers in the warehouse, it is still difficult to meet the demand for efficient and fast picking. In addition, in some scenarios, the unmanned warehouse business of the unmanned retail mode is rapidly rising, and through self-picking by the delivery personnel, it can help the merchant to effectively save labor costs. However, due to the large number of goods in the warehouse and the lack of professional sorting experience of personnel, the self-picking proficiency is not enough, and the efficiency of manual picking is limited, and risks such as mistaken picking and missed picking are prone to occur. Therefore, improving the efficiency and accuracy of goods picking of the delivery personnel or users in the warehouse is a key challenge in warehouse picking operation. SUMMARY

[0003] The purpose of the present disclosure is to provide a picking method using a smart wearable device, a device and an apparatus for warehouse picking.

[0004] According to a first aspect of an embodiment of the present disclosure, a warehouse picking method using a smart wearable device is provided, comprising: obtaining environment data collected by the smart wearable device, the environment data comprising voice data collected by a voice collection device of the smart wearable device and emitted by a delivery personnel in association with goods of a sorting order of the delivery personnel, and image data collected by an image collection device of the smart wearable device and associated with the goods of the sorting order of the delivery personnel; analyzing the voice data to determine a voice instruction of the delivery personnel associated with the sorting order, and identifying the image data to determine goods information of the delivery personnel associated with the sorting order; obtaining, according to at least one of the voice instruction, the goods information, and a behavior stage of the delivery personnel in the target warehouse for picking, guide content information associated with the sorting order output to the delivery personnel wearing the smart wearable device; outputting the guide content information associated with the sorting order through a voice playing mode of the smart wearable device or through an augmented reality projection mode, wherein the augmented reality projection mode enables the delivery personnel to view the guide content information while observing the real scene environment of the goods in the goods location.

[0005] Optionally, the method further comprises: The voice collection device of the smart wearable device collects natural voice of the delivery person aiming at a missing product in the sorting order, identifies and analyzes the natural voice, understands the missing product intention corresponding to the natural voice to enter a missing product feedback mode, determines missing product information of the missing product pointed by the natural voice, and sends the missing product information of the missing product to a sorting and delivery service system.

[0006] Optionally, the method further comprises: The voice collection device of the smart wearable device collects voice data of the delivery person and performs voice recognition to obtain text data corresponding to the voice data. Intention recognition is performed on the text data to obtain an intention recognition result. According to the intention recognition result and a behavior stage of the delivery person in the target warehouse for sorting, guide content information related to the sorting order output to the delivery person wearing the smart wearable device is obtained.

[0007] According to a second aspect of the embodiments of the present disclosure, a smart wearable device for warehouse sorting is provided, comprising: A voice interaction unit is configured to collect multi-modal voice data of the delivery person associated with product of a sorting order of the delivery person, and output synthetic voice corresponding to guide content information related to the sorting order to the delivery person in a voice playing mode. An image collection unit is configured to collect image data associated with product of a sorting order of the delivery person. An image projection unit is configured to output guide content information related to the sorting order to the delivery person in an augmented reality projection mode, wherein the augmented reality projection mode enables the delivery person to watch the guide content information while observing a real scene environment of a product in a storage location.

[0008] According to a third aspect of the embodiments of the present disclosure, a smart device for warehouse sorting is provided, comprising: An information collection module is configured to collect multi-modal voice data of the delivery person associated with product of a sorting order of the delivery person, and collect image data associated with product of a sorting order of the delivery person. An information recognition module is configured to analyze the multi-modal voice data to determine voice instructions of the delivery person related to the sorting order, and recognize the image data to determine product information of the delivery person related to the sorting order. The behavior processing module is configured to invoke a corresponding behavior processing agent to process the voice instruction associated with the sorting order of the delivery personnel, the information of the goods associated with the sorting order of the delivery personnel, and at least one of the behavior stage in which the delivery personnel is currently performing picking in the target warehouse, to obtain guiding content information related to the sorting order output to the delivery personnel wearing the smart wearable device. The information output module is configured to output the guiding content information related to the sorting order to the delivery personnel by voice playing synthesized voice or by an augmented reality projection mode, wherein the augmented reality projection mode enables the delivery personnel to view the guiding content information while observing the real scene environment of the goods location.

[0009] Optionally, the behavior processing module is further configured to invoke a dialogue agent to identify the semantic intention of the delivery personnel based on the voice instruction associated with the sorting order of the delivery personnel, and process the voice instruction associated with the sorting order of the delivery personnel according to the semantic intention, and answer the picking question of the delivery personnel about the sorting order.

[0010] Optionally, the behavior processing module is further configured to, in the case of receiving an order query request, invoke an order query agent to determine the sorting order information of the delivery personnel in the target warehouse according to the order query voice instruction of the delivery personnel, and determine the corresponding guiding content information according to the sorting order information of the delivery personnel in the target warehouse.

[0011] Optionally, the behavior processing module is further configured to, in the case that the current behavior stage is a goods location guiding stage, invoke a goods location guiding agent to determine whether the delivery personnel has arrived at the goods location where the to-be-picked goods are located in the target warehouse according to the goods information determined by the goods location image during the process of the delivery personnel going to the goods location where the to-be-picked goods are located in the target warehouse, and determine the corresponding guiding content information according to the result of whether the delivery personnel has arrived at the goods location where the to-be-picked goods are located in the target warehouse.

[0012] Optionally, the behavior processing module is further configured to, in the case that the current behavior stage is a stage of picking the to-be-picked goods, invoke a picking agent to perform goods state feature recognition on the image of the picked goods according to the image of the picked goods, determine the comparison result of the information of the picked goods of the delivery personnel and the information of the to-be-picked goods according to the goods state feature recognition result, and determine the corresponding guiding content information according to the comparison result of the information of the picked goods of the delivery personnel and the information of the to-be-picked goods.

[0013] By the technical solution, voice data associated with the sorting order goods of the delivery personnel and collected by the voice collection device of the intelligent wearable device, and image data associated with the sorting order goods of the delivery personnel and collected by the image collection device of the intelligent wearable device are acquired. Then, the voice data can be analyzed to determine the voice instruction of the delivery personnel about the sorting order, and the image data can be recognized to determine the goods information of the delivery personnel about the sorting order. Then, the guiding content information related to the sorting order can be output to the delivery personnel wearing the intelligent wearable device according to at least one of the voice instruction, the goods information, and a behavior stage of the delivery personnel in the target warehouse for order picking. Finally, the guiding content information related to the sorting order can be output through the voice playing mode of the intelligent wearable device or through the augmented reality projection mode. Since the guiding content information related to the sorting order can be output through the voice playing mode of the intelligent wearable device or through the augmented reality projection mode to guide the order picking process of the delivery personnel, the difficulty of order picking of the delivery personnel can be reduced, and the order picking efficiency can be improved.

[0014] Other features and advantages of the present disclosure will be described in detail in the following detailed description section. BRIEF DESCRIPTION OF DRAWINGS

[0015] The accompanying drawings are included to provide a further understanding of the present disclosure and constitute a part of the specification, which together with the following detailed description, serve to explain the present disclosure. In the drawings: Figure 1 is an architecture schematic diagram of an unmanned warehouse order picking system according to an exemplary embodiment; Figure 2 is a flowchart of an order picking method using an intelligent wearable device according to an exemplary embodiment; Figure 3 is a schematic diagram of order picking information output on a display interface of AR intelligent glasses according to an exemplary embodiment; Figure 4 is a flowchart of a normal process in a picking scenario of a warehouse order picking method using an intelligent wearable device according to an exemplary embodiment; Figure 5 is a schematic diagram of a system architecture and related data flow according to an exemplary embodiment; Figure 6 is a process schematic diagram of an intelligent glasses device worn by a user and a server interaction according to an exemplary embodiment; Figure 7 is a structural block diagram of an intelligent wearable device for warehouse order picking according to an exemplary embodiment; Figure 8 is a structural block diagram of an AR device for warehouse picking according to an exemplary embodiment. DETAILED DESCRIPTION

[0016] The specific embodiments of the present disclosure are described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to illustrate and explain the present disclosure, and are not intended to limit the present disclosure.

[0017] It should be noted that all actions of obtaining signals, information or data in the present disclosure are carried out in accordance with the corresponding data protection regulations and policies of the country where the device is located, and with the authorization of the corresponding device owner.

[0018] Figure 1 A structural schematic diagram of an unmanned warehouse picking system according to an embodiment of the present disclosure is shown. As shown in the figure, the unmanned warehouse picking system includes a delivery person 101, an intelligent wearable device 102, a warehouse 103 (which can be an unmanned warehouse or a manned warehouse), and a server 104. The intelligent wearable device 102 can be an intelligent head-mounted device, such as AR smart glasses, a smart helmet, etc. The server 104 can be a collection of servers with multiple functions, which can be a single server or a server cluster. For example, the server 104 can include an order management server, a picking and delivery management server (delivery management server), a warehouse management server, etc. The delivery person 101 can wear the intelligent wearable device 102 to pick in the unmanned warehouse 103. Through the cooperation of the server 104 and the intelligent wearable device 102, the process of the delivery person 101 picking in the unmanned warehouse 103 can be managed and assisted. Figure 1

[0019] In some embodiments, the intelligent wearable device 102 can communicate with the server 104, adopt an AI Agent development mode, have intelligent AI functions, and through the multi-modal recognition and interaction capabilities of AIGC, intelligently recognize goods, shelves, and process demands, etc. to accurately complete goods warehousing and shelving, and warehouse picking operations in real time. The overall architecture is divided into four layers: 1. Interaction layer: provides voice and video stream input data of the user for the Agent, and is also responsible for issuing the input of the Agent to the user.

[0020] 2. Agent layer: mainly includes three Agents, namely ChatAgent, PickerAgent and ErrorAgent.

[0021] 3. AI service layer: provides basic large model capability encapsulation, tool encapsulation, memory storage, and multi-modal data perception.

[0022] ​4. Infrastructure layer: including a large model development framework, a large model interface and a database.

[0023] Figure 2 FIG. 1 is a flowchart of a picking method using a smart wearable device according to an example embodiment, as shown in FIG. 1, the picking method using a smart wearable device can include the following steps: Figure 2 S210, obtaining environment data collected by the smart wearable device, the environment data including voice data associated with the picking order goods of the delivery personnel collected by the voice collection device of the smart wearable device, and image data associated with the picking order goods of the delivery personnel collected by the image collection device of the smart wearable device.

[0024] In the embodiments of the present disclosure, the delivery personnel can wear the smart wearable device to the warehouse to pick according to the picking order. During the picking process, the delivery personnel can issue voice data associated with the picking order goods as needed, such as voice data for querying the picking order goods information, so that the voice data associated with the picking order goods of the delivery personnel issued by the voice collection device of the smart wearable device. In addition, during the picking process, the image collection device of the smart wearable device can also collect image data associated with the picking order goods of the delivery personnel.

[0025] In some embodiments, the picking order can be associated with the delivery end account logged in the smart wearable device.

[0026] S220, analyzing the voice data to determine the voice instructions of the delivery personnel associated with the picking order, and identifying the image data to determine the goods information of the delivery personnel associated with the picking order.

[0027] After obtaining the voice data associated with the picking order goods of the delivery personnel issued by the delivery personnel, the voice data can be analyzed to obtain the voice instructions of the delivery personnel associated with the picking order. After obtaining the image data associated with the picking order goods of the delivery personnel issued by the delivery personnel, the image data can be identified to obtain the goods information associated with the picking order.

[0028] In some embodiments, the voice data can be analyzed by the information identification module in the smart wearable device to determine the voice instructions of the delivery personnel associated with the picking order, and the image data can be identified to determine the goods information of the delivery personnel associated with the picking order.

[0029] S230, obtaining the guidance content information related to the picking order output to the delivery personnel wearing the smart wearable device according to at least one of the voice instructions, the goods information, and the behavior stage of the delivery personnel currently picking in the target warehouse.

[0030] ​In the embodiments of the present disclosure, the guidance content information is used to guide the delivery personnel to successfully complete the picking process in the warehouse and manage the picking process, thereby improving the picking efficiency of the delivery personnel.

[0031] In the embodiments of the present disclosure, the content of the guidance content information related to the sorting order obtained under the combination of one or more of the voice instruction, the goods information and the behavior stage is different.

[0032] In S240, the guidance content information related to the sorting order is output by a voice playing mode of the smart wearable device or by an augmented reality projection mode, wherein the augmented reality projection mode makes the delivery personnel watch the guidance content information while observing the real scene environment of the goods in the goods location.

[0033] In the embodiments of the present disclosure, after obtaining the guidance content information related to the sorting order, the smart wearable device can output the guidance content information related to the sorting order to the delivery personnel by a voice playing mode or by an augmented reality projection mode.

[0034] The augmented reality projection mode refers to a display mode in which virtual information or images are superimposed into the real-time field of view of a user. Optionally, a generated image can be guided to the eyes of the user through a special optical assembly, such as a waveguide, to realize the augmented reality projection. Optionally, the generated image can be transmitted to the front of the glasses through a medium by light conduction to realize the augmented reality projection. Optionally, a semi-transparent lens can be used to make the light of the generated image and external light enter the eyes together to realize the augmented reality projection.

[0035] For example, as shown in Figure 3 When the smart wearable device outputs the guidance content information related to the sorting order to the delivery personnel by the augmented reality projection mode, the optical assembly in the smart wearable device can transmit the image of the guidance content information to the eyes or the front of the eyes of the delivery personnel wearing the smart wearable device by light projection or light reflection, so that the delivery personnel can simultaneously observe the real scene environment of the goods in the goods location and the guidance content information.

[0036] With the above method, the voice data associated with the sorting order goods of the delivery personnel issued by the voice collection device of the smart wearable device and the image data associated with the sorting order goods of the delivery personnel collected by the image collection device of the smart wearable device are obtained, then the voice data associated with the sorting order of the delivery personnel is determined by analyzing the voice data, the goods information associated with the sorting order of the delivery personnel is determined by identifying the image data, then the guiding content information related to the sorting order output to the delivery personnel wearing the smart wearable device is obtained according to at least one of the voice instruction, the goods information and the behavior stage of the delivery personnel currently picking goods in the target warehouse, and finally the guiding content information related to the sorting order can be output through the voice playing mode of the smart wearable device or through the augmented reality projection mode. Since the guiding content information related to the sorting order can be output through the voice playing mode of the smart wearable device or through the augmented reality projection mode to guide the picking process of the delivery personnel, the difficulty of the delivery personnel in picking goods can be reduced, and thus the picking efficiency can be improved.

[0037] It should be noted that the data used in the smart wearable device picking method of the embodiments of the present disclosure can be obtained by the smart wearable device through direct or indirect means. Direct acquisition includes but is not limited to self-collection or calculation of data, and indirect acquisition includes but is not limited to data transmission between other devices.

[0038] In some embodiments, obtaining the guiding content information related to the sorting order output to the delivery personnel wearing the smart wearable device according to at least one of the voice instruction, the goods information and the behavior stage of the delivery personnel currently picking goods in the target warehouse can include the following steps: understanding the semantic intention of the voice data of the delivery personnel collected by the smart wearable device, and generating the answer information corresponding to the voice data of the delivery personnel and associated with the current sorting action of the delivery personnel according to at least one of the behavior stage of the delivery personnel currently picking goods in the target warehouse, the scene image information associated with the sorting action of the delivery personnel determined by the image data collected by the smart wearable device, and the goods information; the voice playing device of the smart wearable device outputs the answer information.

[0039] In the embodiments of the present disclosure, the smart wearable device can understand the semantic intention of the voice data of the delivery personnel, and generate the answer information corresponding to the voice data of the delivery personnel and associated with the current sorting action of the delivery personnel in combination with other collected information, to answer the voice data of the delivery personnel.

[0040] In the embodiments of the present disclosure, the semantic intention of the voice data of the delivery terminal can include an order query intention, a picking instruction intention, and a picking problem reporting intention. The order query intention can include, for example, a to-be-picked order list query intention and a certain to-be-picked order detail query intention. The picking instruction intention can include, for example, an instruction intention for starting picking a certain order. The picking problem reporting intention can include, for example, a shortage reporting intention.

[0041] For example, the delivery staff issues a voice: Where is a certain product? The smart wearable device (glasses or a helmet) answers: On the first shelf.

[0042] For another example, the delivery staff issues a voice: Cancel xxx orders, shortage. The smart wearable device answers: Canceled for you.

[0043] For another example, the delivery staff issues a voice: Pick the second to-be-picked order in the order list. The smart wearable device answers: The second to-be-picked order details have been output to you in an augmented reality projection manner, and the path information for reaching the second to-be-picked order has been output to you. Please check.

[0044] For another example, the delivery staff issues a voice: The quantity is insufficient. Report a shortage, and he holds up the goods in front of the smart wearable device and shakes them. The smart wearable device determines the intention according to the voice, which is to report a shortage. In addition, the smart wearable device identifies the product information according to the image of the goods. The glasses reply: OK, the quantity of the product is insufficient, and the insufficient order has been canceled.

[0045] For another example, the delivery staff issues a voice: Where is a certain product? The smart wearable device determines that the delivery staff is currently at the third shelf according to the collected image data, and then can answer: On the first shelf, go left to reach it. The delivery staff continues to issue a voice: How many of this product are left? The smart wearable device can answer: There are 5 left, and your picking quantity is 3, which meets your picking demand, after querying the picking order information of the delivery staff.

[0046] For example, the deliveryman issues a voice query: query the current list of orders to be picked up; the smart wearable device answers: OK, I'm helping you query. After querying the result, the smart wearable device continues to answer: output through augmented reality projection, please check. The deliveryman continues to issue a voice query: query the second order to be picked up in the list of orders to be picked up; the smart wearable device answers: OK, I'm helping you query. After querying the result, the smart wearable device continues to answer: output the second order to be picked up through augmented reality projection, and the path information to the second order to be picked up, please check. The deliveryman continues to issue a voice query: forget it, I want to pick up the third order to be picked up in the list of orders to be picked up; the smart wearable device answers: OK, I'm helping you query the third order to be picked up. After querying the result, the smart wearable device continues to answer: output the third order to be picked up through augmented reality projection, and the path information to the third order to be picked up, please check.

[0047] As can be seen, the embodiments of the present disclosure can query the voice data of the deliveryman according to the collected multi-modal information (i.e., voice data and image data), and the voice data of the deliveryman can be output through voice playing or augmented reality projection.

[0048] That is, in some embodiments, the process of understanding the semantic intention of the voice data of the deliveryman collected by the smart wearable device, and generating answer information corresponding to the voice data of the deliveryman and associated with the current sorting action of the deliveryman according to at least one of the current behavior stage of the deliveryman in the target warehouse, the scene image information related to the sorting action of the deliveryman determined by the image data collected by the smart wearable device, and the goods information can be a multi-round interaction process, so that the deliveryman and the smart wearable device can perform multi-round voice interaction or multi-round question and answer.

[0049] In some embodiments, the method for picking up goods using a smart wearable device according to the embodiments of the present disclosure can further include the following steps: receiving a query voice of the deliveryman for the location of the goods to be sorted; After voice recognition and semantic analysis, it is determined that the intention is to query the goods, and the agent calls the order management module to find the goods storage record of the target warehouse corresponding to the goods information of the order to be sorted based on the goods information of the order to be sorted, and queries the location of the goods to be sorted; The voice playing device of the smart wearable device outputs the location of the goods to be sorted.

[0050] Similarly, the deliveryman can also output a voice with other semantic intentions, and the smart wearable device can output corresponding answer information and output it through the voice playing device.

[0051] In some embodiments, obtaining the guidance content information related to the picking order to be output to the delivery personnel wearing the smart wearable device according to at least one of the voice instruction, the item information, and the behavior stage of the delivery personnel in the process of picking in the target warehouse can include the following steps: In the case that the current behavior stage is the location guiding stage, determining whether the delivery personnel wearing the smart wearable device reaches the location of the item to be picked in the target warehouse based on the item information determined by the smart wearable device from the location images collected by the smart wearable device in the process of the delivery personnel going to the location of the item to be picked in the target warehouse, the item to be picked being the item in the picking order corresponding to the delivery personnel in the target warehouse; In the case that the delivery personnel wearing the smart wearable device reaches the location of the item to be picked, determining that the guidance content information includes at least one of the picking voice prompt or the product detail text information of the item to be picked; and / or In the case that the delivery personnel wearing the smart wearable device does not reach the location of the item to be picked, determining that the guidance content information includes the picking guidance information, the picking guidance information being used to guide the delivery personnel to go to the location of the item to be picked in the target warehouse, the picking guidance information including at least one of the voice prompt guiding the moving direction of the delivery personnel, the path image guiding the moving direction of the delivery personnel, or the display image of the item to be picked.

[0052] In the embodiments of the present disclosure, the delivery personnel can follow the picking guidance information in the guidance content information to go to the location of the item to be picked in the target warehouse, and this process stage can be understood as the location guiding stage. If the current behavior stage of the delivery personnel in the target warehouse is the location guiding stage, the smart wearable device can determine whether the delivery personnel reaches the location of the item to be picked in the target warehouse based on the item information determined by the smart wearable device from the location images collected by the smart wearable device in the process of the delivery personnel going to the location of the item to be picked in the target warehouse, and further obtain different guidance content information according to whether the delivery personnel reaches the location of the item to be picked.

[0053] In the embodiments of the present disclosure, if it is determined that the delivery personnel wearing the smart wearable device reaches the location of the item to be picked, it can be determined that the guidance content information includes at least one of the picking voice prompt or the product detail text information of the item to be picked. Here, the picking voice prompt can be, for example, “arrive at the specified location and start picking”, and the product detail text information of the item to be picked can include, for example, at least one of the product name, the shelf number, and the product quantity.

[0054] In the embodiments of the present disclosure, if it is determined that the delivery personnel wearing the smart wearable device has not arrived at the location of the to-be-picked goods, the guidance content information can include at least one of the picking guidance information, such as a voice prompt guiding the moving direction of the delivery personnel, a path image guiding the moving direction of the delivery personnel, or a display image of the to-be-picked goods, so as to continue to guide the delivery personnel to the location of the to-be-picked goods in the target warehouse. Here, the path image guiding the moving direction of the delivery personnel can be output by the augmented reality projection mode of the smart wearable device.

[0055] By using the above method, in the case that the current behavior stage is the location guiding stage, the progress of the delivery personnel can be gradually guided, and the delivery personnel is reminded to pick the goods when the delivery personnel arrives at the location of the to-be-picked goods in the target warehouse. Therefore, the difficulty of the delivery personnel in searching for the way and tracking the location can be reduced, and the picking efficiency and experience of the delivery personnel can be improved.

[0056] In some embodiments, determining whether the delivery personnel arrives at the location of the to-be-picked goods in the target warehouse based on the goods information determined based on the location images of the delivery personnel in the process of the delivery personnel going to the location of the to-be-picked goods in the target warehouse can include the following steps: performing location environment feature recognition on the location images of the delivery personnel in the process of the delivery personnel going to the location of the to-be-picked goods in the target warehouse to obtain a location environment feature recognition result, the to-be-picked goods being the goods in the sorting order corresponding to the delivery personnel in the target warehouse; determining that the delivery personnel arrives at the location of the to-be-picked goods in the target warehouse in a case where it is determined based on the location environment feature recognition result that the location images of the delivery personnel in the process of the delivery personnel going to the location of the to-be-picked goods in the target warehouse exist in a location environment consistent with the location of the to-be-picked goods.

[0057] In the embodiments of the present disclosure, the goods information can be the location environment feature recognition result, that is, the recognized location environment. Alternatively, the location environment can include a location environment image or a location identifier.

[0058] In a case where the location environment feature recognition result includes a location environment image, it can be determined whether the location environment image in the location environment feature recognition result is consistent with the location environment image of the location of the to-be-picked goods. If the location environment image in the location environment feature recognition result is consistent with the location environment image of the location of the to-be-picked goods, it is determined that the delivery personnel arrives at the location of the to-be-picked goods.

[0059] In a case where the location environment feature recognition result includes a location identifier, it can be determined whether the location identifier in the location environment feature recognition result is consistent with the identifier of the location of the to-be-picked goods. If the location identifier in the location environment feature recognition result is consistent with the identifier of the location of the to-be-picked goods, it is determined that the delivery personnel arrives at the location of the to-be-picked goods.

[0060] That is, in the embodiments of the present disclosure, the image of the goods location in the process of the delivery person going to the goods location where the to-be-picked goods are located in the target warehouse can be collected to identify the goods location environment features, so as to prompt whether the delivery person arrives at the goods location where the to-be-picked goods are located according to the identification result of the goods location environment features, so as to further improve the efficiency of the delivery person picking goods in the target warehouse.

[0061] In some embodiments, the goods location environment features of the image of the goods location in the process of the delivery person going to the goods location where the to-be-picked goods are located in the target warehouse can be identified by a pre-trained goods location environment feature identification model to obtain the goods location environment feature identification result.

[0062] In some embodiments, based on the goods information determined by the image of the goods location in the process of the delivery person going to the goods location where the to-be-picked goods are located in the target warehouse collected by the intelligent wearable device, whether the delivery person arrives at the goods location where the to-be-picked goods are located in the target warehouse can include the following steps: In the case that the current behavior stage is the stage of picking the to-be-picked goods, the image acquisition device of the intelligent wearable device of the delivery person acquires the image of the picked goods, identifies the goods state features of the picked goods to obtain the goods state feature identification result, and the to-be-picked goods are the goods in the sorting order corresponding to the delivery person in the target warehouse; In the case that the comparison result of the information of the picked goods and the information of the to-be-picked goods determined according to the goods state feature identification result is matching, it is determined that the guiding content information includes at least one of the voice prompt that the to-be-picked goods picking is completed output by the voice playing of the intelligent wearable device or the picking prompt of the next to-be-picked goods output to the delivery person by the augmented reality projection; or In the case that the comparison result of the information of the picked goods and the information of the to-be-picked goods determined according to the goods state feature identification result is not matching, it is determined that the guiding content information includes at least one of the voice prompt that the information of the picked goods does not match the information of the to-be-picked goods output by the voice playing of the intelligent wearable device or the prompt that the information of the picked goods does not match the information of the to-be-picked goods output to the delivery person by the augmented reality projection.

[0063] In the embodiments of the present disclosure, the delivery person can arrive at the goods location where the to-be-picked goods are located in the target warehouse, and the process stage after arriving at the goods location where the to-be-picked goods are located in the target warehouse and before completing picking can be understood as the stage of picking the to-be-picked goods.

[0064] If the behavior stage of the delivery staff currently in the target warehouse is the goods location guiding stage, the intelligent wearable device can perform goods state feature recognition on the image of the picked goods based on the collected image of the picked goods, obtain a goods state feature recognition result, and determine whether the comparison result of the information of the picked goods and the information of the to-be-picked goods matches according to the goods state feature recognition result, and further obtain different guiding content information according to the comparison result.

[0065] If the comparison result of the information of the picked goods and the information of the to-be-picked goods is determined to be matched according to the goods state feature recognition result, it means that the delivery staff picks correctly, and then at least one of the following can be obtained: a voice prompt of the to-be-picked goods picking completion output by the voice playing of the intelligent wearable device, or a picking prompt of the next to-be-picked goods output to the delivery staff by the augmented reality projection.

[0066] If the comparison result of the information of the picked goods and the information of the to-be-picked goods is determined to be mismatched according to the goods state feature recognition result, it means that the delivery staff picks incorrectly, and then at least one of the following can be obtained: a voice prompt that the information of the picked goods does not match the information of the to-be-picked goods output by the voice playing of the intelligent wearable device, or a prompt that the information of the picked goods does not match the information of the to-be-picked goods output to the delivery staff by the augmented reality projection, so as to prompt the delivery staff that the picking is incorrect.

[0067] In some embodiments, the next to-be-picked goods can be determined according to the goods name or goods type in the picking order.

[0068] In some embodiments, the image of the picked goods can be recognized by a pre-trained goods state feature recognition model to obtain the goods state feature recognition result of the image of the picked goods.

[0069] In some embodiments, when all the goods in the picking order are picked, and the comparison result of the information of each picked goods and the information of the corresponding to-be-picked goods is determined to be matched based on the goods state feature recognition result of each picked goods, it can be determined that the guiding content information includes a voice prompt of the picking completion in the target warehouse output by the voice playing of the intelligent wearable device, or a prompt information of the picking completion in the target warehouse output to the delivery staff by the augmented reality projection.

[0070] In some embodiments, the information of the to-be-picked product includes at least one of a to-be-picked product name and a to-be-picked product quantity, in which case, it can be determined whether the product name in the product state feature recognition result of the image of the picked product matches the to-be-picked product in the information of the to-be-picked product, and / or, in the case where the product name in the recognition result of the image of the picked product matches the to-be-picked product in the information of the to-be-picked product of the picking order, whether the product quantity in the recognition result of the image of the picked product matches the product quantity of the to-be-picked product in the information of the to-be-picked product of the picking order.

[0071] In the case where the information of the to-be-picked product includes a to-be-picked product name, the image of the picked product can be recognized to obtain a recognition result including a product name, and then it can be directly determined whether the product name in the product state feature recognition result of the image of the picked product matches the to-be-picked product in the information of the to-be-picked product, and if so, it can be determined that the comparison result of the information of the picked product of the delivery personnel and the information of the to-be-picked product matches, otherwise, it can be determined that the comparison result of the information of the picked product of the delivery personnel and the information of the to-be-picked product does not match.

[0072] In the case where the information of the to-be-picked product includes a to-be-picked product name and a to-be-picked product quantity, the image of the picked product can be recognized to obtain a recognition result including a product name and a product quantity, and then it can be first determined whether the product name in the product state feature recognition result of the image of the picked product matches the to-be-picked product in the information of the to-be-picked product, and if so, it can be further determined whether the product quantity in the product state feature recognition result of the image of the picked product matches the product quantity of the to-be-picked product in the information of the to-be-picked product, and if so, it can be determined that the comparison result of the information of the picked product of the delivery personnel and the information of the to-be-picked product matches.

[0073] In some embodiments, the method for picking using a smart wearable device according to the embodiments of the present disclosure can further include the following steps: recognizing the picking instruction included in the voice data, and determining that the behavior stage of the delivery personnel enters the position guiding stage; obtaining a sorting order associated with the delivery end account corresponding to the smart wearable device, determining the position of the to-be-picked product in the target warehouse according to the storage record of the to-be-picked product in the target warehouse in the sorting order, and determining the picking guidance information of the to-be-picked product in the target warehouse in the sorting order to be output to the delivery personnel based on the current position of the delivery personnel wearing the smart wearable device and the position of the to-be-picked product in the target warehouse in the sorting order; outputting the picking guidance information to the delivery personnel through the voice playing mode of the smart wearable device or through the augmented reality projection mode.

[0074] In the disclosed embodiment, a delivery person may come to a target warehouse to pick up goods. In this scenario, the delivery person may issue an instruction to pick up goods from the target warehouse. For example, the delivery person may say "XXX, I'm going to start picking up goods" via voice. Thus, the voice collection device of the smart wearable device may collect the voice data issued by the delivery person and recognize the picking instructions included in the voice data. If the picking instructions included in the voice data are recognized, it can be determined that the delivery person's behavior phase has entered the cargo location guidance phase. Then, the smart wearable device may begin to generate picking guidance information for the goods to be picked in the sorting order in the target warehouse, which is output to the delivery person, so as to guide the delivery person to the cargo location in the target warehouse where the goods to be picked are located.

[0075] In the disclosed embodiment, a sorting order associated with the delivery-end account corresponding to the smart wearable device can be obtained, and the items to be picked can be determined from the sorting order. Then, based on the corresponding shelf information of the items to be picked in the warehouse recorded in the delivery service system (for example, the system backend records "paper towels are on shelf xx, number xx"), the warehouse location of the items to be picked in the picking order information can be queried. Subsequently, the current location of the delivery person wearing the smart wearable device can be further collected, and based on the delivery person's current location and the location of the items to be picked in the sorting order in the target warehouse, the picking guidance information for the items to be picked in the sorting order in the target warehouse can be determined and output to the delivery person.

[0076] In some implementations, the method for picking goods using a smart wearable device according to an embodiment of the present disclosure may further include the following steps: Obtain information on multiple items included in the sorting order, determine the picking order for the multiple items, determine information on items to be picked based on the picking order, output the information on items to be picked through smart wearable devices, and guide delivery personnel to pick items at the target storage locations corresponding to the items in the order of the output information on items to be picked.

[0077] In the embodiment of the present disclosure, a sorting order may include multiple goods, and each good has corresponding product information. Furthermore, the picking order of the multiple goods can be determined based on the multiple product information included in the sorting order, and the information of the goods to be picked can be further determined based on the picking order, so as to guide the delivery staff to pick the goods at the target storage location corresponding to the goods in the order of the output information of the goods to be picked.

[0078] In the disclosed embodiment, the order for picking multiple items may be, for example, picking item A first, then picking item B second, and so on until the last item in the sorting order is reached. The item information to be picked can be understood as information about a specific item that the delivery person is currently picking, rather than information about multiple items.

[0079] In some embodiments, the item information includes an item name or an item type, and then the picking sequence of the plurality of items can be determined according to the item names and / or the item types of the plurality of items.

[0080] In some embodiments, the picking method using the smart wearable device of the present disclosure can further include the following steps: The voice collection device of the smart wearable device collects the natural voice of the delivery personnel aiming at the out-of-stock item in the sorting order, recognizes and analyzes the natural voice, understands the out-of-stock intention corresponding to the natural voice to enter the out-of-stock feedback mode, determines the out-of-stock information of the out-of-stock item pointed by the natural voice, and sends the out-of-stock information of the out-of-stock item to the picking and delivery service system.

[0081] In the present disclosure, when the delivery personnel arrives at the location of the out-of-stock item after picking, it may be found that the out-of-stock item is out of stock. In this scenario, the delivery personnel can issue a natural voice aiming at the out-of-stock item, for example, the delivery personnel can say "no stock, how to pick" by voice, so that the voice collection device of the smart wearable device can collect the natural voice issued by the delivery personnel, recognize and analyze it, understand the out-of-stock intention corresponding to the natural voice to enter the out-of-stock feedback mode, determine the out-of-stock information of the out-of-stock item pointed by the natural voice, and then the smart wearable device can send the out-of-stock information of the out-of-stock item to the picking and delivery service system.

[0082] It should be noted that the out-of-stock information here can include two kinds of out-of-stock information: no out-of-stock item, or the number of out-of-stock items is insufficient.

[0083] For the smart wearable device, there is a process of understanding the intention of the natural voice, involving NLP intelligent understanding, understanding the language to start the out-of-stock mode (entering the out-of-stock feedback mode), and understanding the out-of-stock object referred by the language.

[0084] As can be seen, in the present disclosure, a function of reporting the out-of-stock information of the out-of-stock item by the delivery personnel when the out-of-stock item is out of stock is provided, which is convenient for helping the operation and maintenance personnel to manage the items in the target warehouse.

[0085] In some embodiments, the picking method using the smart wearable device of the present disclosure can further include the following steps: The voice collection device of the smart wearable device collects the voice data of the delivery personnel and performs voice recognition to obtain text data corresponding to the voice data; Intention recognition is performed on the text data to obtain an intention recognition result; According to the intention recognition result and the behavior stage of the delivery personnel currently picking in the target warehouse, guide content information related to the sorting order output to the delivery personnel wearing the smart wearable device is obtained.

[0086] In the embodiments of the present disclosure, in addition to being able to speak voice instructions, the delivery personnel can also speak descriptive voice. In this case, the voice collection device of the smart wearable device can collect voice data of the delivery personnel and perform voice recognition to obtain text data corresponding to the voice data, and then further perform intent recognition on the text data to obtain an intent recognition result. Subsequently, the guiding content information related to the order picking that is output to the delivery personnel wearing the smart wearable device can be obtained according to the intent recognition result and the current behavior stage of the delivery personnel in the target warehouse.

[0087] For example, the descriptive voice can be "I have arrived at XX storage location". If the current behavior stage of the delivery personnel in the target warehouse is the stage of picking the to-be-picked goods, the smart wearable device can determine that the delivery personnel has arrived at the storage location where the to-be-picked goods are located in the target warehouse, and then can determine that the guiding content information includes at least one of the picking voice prompt or the text information of the product details of the to-be-picked goods.

[0088] That is, in the embodiments of the present disclosure, in addition to being able to automatically recognize whether a certain condition is reached by collecting image data through the image collection device of the smart wearable device, the voice data output by the user in different behavior stages can also be used to assist the smart wearable device in determining whether a certain condition is reached, thereby improving the understanding ability of the smart wearable device for environmental conditions.

[0089] The picking scenario of the above-mentioned warehouse picking method using a smart wearable device will be described exemplarily as follows: Normal process: The target delivery personnel or target user (i.e. the delivery personnel) arrives at the warehouse and wears the AR smart glasses. The target delivery personnel or target user who logs in through the AR smart glasses obtains the current target delivery personnel or target user information through the target delivery personnel or target user account. The Agent (intelligent agent) pulls the to-be-picked order of the current target delivery personnel or target user, and displays the order information (i.e. the order picking information). The target delivery personnel or target user is guided to start picking from the first goods: the target delivery personnel or target user is guided to a certain storage location of a certain shelf for picking through voice prompts and simple illustrations. During the picking process, the taken goods or the storage location number is automatically captured and recognized, and whether the taken goods quantity is correct is determined. If there is an abnormal situation, a voice prompt will be given to inform the target delivery personnel or target user whether the goods are taken incorrectly.

[0090] Abnormal process: If the target delivery personnel or target user encounters a situation of insufficient goods quantity during picking, some specific complaint language of the target delivery personnel or target user or empty storage location image is used. The Agent recognizes this scenario, and at the same time triggers a workflow to perform a cancellation operation, informs the target delivery personnel or target user that the current order cannot be completed, needs to be cancelled, and notifies the user, and initiates a task of checking the order.

[0091] In the above warehouse picking method using a smart wearable device, the AI technology is combined with the wearable device to realize the ability of "smart device as warehouse management", thereby greatly reducing the operation and maintenance cost of manual warehouse management.

[0092] In addition, the AR smart glasses can also monitor the picking operation process in real time and timely remind errors to ensure accurate operation.

[0093] The replenishment scene of the above warehouse picking method using a smart wearable device is exemplarily described as follows: Normal process: The operation and maintenance personnel enter the replenishment process through voice. The smart glasses control the replenishment process, prompt the personnel to pick up the goods, identify, and then prompt which storage location to replenish. During the process of the operation and maintenance personnel putting the goods into the storage location, whether the goods are put in the correct position is automatically identified. If the goods are put in the wrong position, a voice prompt is given. The replenishment process is repeated until all the goods are completed, and then a replenishment completion prompt is given. For example, the smart glasses can identify the goods name and quantity of the replenishment, automatically record whether the replenishment list is completed, and then give a completion prompt.

[0094] Abnormal process: In the replenishment process, the operation of the operation and maintenance personnel is monitored, the scene of putting the goods in the wrong storage location is identified, and a prompt and correction are given. Here, the correction means that the operation and maintenance personnel can also be told where the goods should be put in the correct position.

[0095] In the above warehouse replenishment method using a smart wearable device, the AI technology is combined with the wearable device to realize the ability of "smart device as warehouse management", thereby greatly reducing the operation and maintenance cost of manual warehouse management. In addition, the AI Agent guides the warehouse management personnel to replenish, improves the picking efficiency, and the smart glasses can also monitor the replenishment operation process in real time, timely remind errors, and ensure accurate operation.

[0096] In the embodiments of the present disclosure, through the multi-modal recognition and interaction ability of AIGC, the goods, shelves, and process demands are intelligently identified, and the goods warehousing and shelving, storage location picking, and other operations are accurately completed in real time, so that a set of AR+AI smart hardware devices realize the operation and management of a low-cost unmanned warehouse.

[0097] Next, the above method for picking goods in a warehouse using a smart wearable device is described in combination with the following Figure 4The normal flow in the picking scene of the above warehouse picking method using the smart wearable device is exemplarily illustrated, which is realized by the interaction of the delivery person end and the device end, wherein the device end includes chatAgent (dialogue agent), pickService (picking service), context (context service), guidAgent (guidance agent), and pickAgent (picking agent) units. Each unit included in the device end can be provided by at least one of the smart wearable device or the server. As shown in Figure 4 The delivery person issues a voice dialogue: Help me start picking.

[0098] The dialogue agent generates a dialogue: Recognize the "picking" instruction, query the current state from the picking service, and create a picking order for the picking service.

[0099] The picking service returns all the picking order information to the dialogue agent.

[0100] The dialogue agent arranges the picking sequence, and outputs the dialogue through the voice playback mode of the smart wearable device: Picking has started, and outputs the picking order information and the first to-be-picked commodity through the augmented reality projection mode. In addition, the dialogue agent also sends a request to the guidance agent to start guiding to the X shelf of the X position.

[0101] The dialogue agent receives the information of the successful acceptance sent by the guidance agent, and outputs the dialogue through the voice playback mode of the smart wearable device: Start shelf guidance.

[0102] The guidance agent judges in real time whether the delivery person has arrived, and if not, sends the guidance information to the dialogue agent: Dialogue: Please move in which direction, data display: Target position picture.

[0103] The dialogue agent outputs the dialogue through the voice playback mode of the smart wearable device: Please move in which direction, and outputs the data display through the augmented reality projection mode: Target position picture.

[0104] The guidance agent judges that the delivery person has arrived at the position, and sends a message to the dialogue agent that the position has been arrived.

[0105] The dialogue agent records the arrival to the context service, and outputs the dialogue through the voice playback mode of the smart wearable device: Arrived at the specified position, please start picking, and outputs the data display through the augmented reality projection mode: To-be-picked commodity details, and sends the information of starting picking to the picking agent.

[0106] The picking agent judges the picking result in real time, and sends the picking prompt "dialogue: XXX" to the dialogue agent according to the picking result. ​

[0107] The dialogue agent outputs the dialogue through the voice playing mode of the smart wearable device: XXX.

[0108] If the picking agent determines that the picking is successful according to the picking result, the picking agent sends a message of picking success to the dialogue agent.

[0109] The dialogue agent updates the picking order information, and outputs the dialogue through the voice playing mode of the smart wearable device: the picking of XX has been completed, please proceed to the next according to the prompt, and outputs the data display through the augmented reality projection mode: picking order details (to-be-picked goods-1).

[0110] As can be seen, the above Figure 4 shows the normal process of the delivery man picking the first to-be-picked goods in the warehouse. In the above process, the delivery man and the smart wearable device can have one or more rounds of question and answer. And in the process of question and answer, the smart wearable device can generate a targeted answer of voice data according to the collected multi-modal data of the delivery man.

[0111] The warehouse picking method using the smart wearable device according to the embodiments of the present disclosure uses the multi-modal recognition and interaction capability of AIGC to intelligently recognize goods, shelves, and process demands, etc., to accurately complete the operation of picking goods and guiding the location, etc., to achieve the operation and management of a low-cost unmanned warehouse by using a set of AR+AI intelligent hardware devices.

[0112] Next, a system architecture diagram and related data flow of the embodiments of the present disclosure will be described. As Figure 5 shown: Figure 5 The AR smart glasses can send picture streams and chat content to the AUK IoT platform, and the AUK IoT platform can send instructions to start video streams, chat replies, and order information displays to the AR smart glasses. The AUK IoT platform and the AR smart glasses service have uplink and downlink information links, and the AR smart glasses service is deployed with an AI agent.

[0113] The AR smart glasses service can send pictures to the database for storage, and obtain picking orders and order information in the database.

[0114] In some embodiments, the AI agent deployed by the AR smart glasses service can include a dialogue agent, an order management agent, a shelf guide agent, a picking agent, and an exception handling agent. Among them:

[0115] ​The conversation agent is used to identify the user's intention. If a custom action is matched, it will be processed by the corresponding agent. The custom action list is as follows: a. Get the current list of pending orders; b. Get the details of a pending order; c. Start picking for an order; d. Problems encountered in picking goods.

[0116] Order Management Agent: Input: Several fixed instructions: 1. Query the current person's pending orders, 2. Query the details of a certain order; 3. Process abnormal orders.

[0117] Output: corresponding data content.

[0118] Shelf Guidance Agent: Memory: Shelf layout in the warehouse (picture or description); Input: target shelf number + location number + picture from the current viewing angle; Output: Whether the target position is reached.

[0119] Picking Agent: Input: the ID of the item to be picked, its feature description, shelf location, context, and n screenshots of the user's current field of view. Output: Whether the picked items are correct. Whether the picking is completed.

[0120] Exception handling agent: Input: out-of-stock information of the picked goods; Output: Cancel or modify the order, inform the user, and notify the merchant to restock.

[0121] Next, combine Figure 6 The process of interaction between a user-worn smart glasses device and a server in an embodiment of the present disclosure is described. Figure 6 As shown: On the one hand, the smart glasses device can record the user's voice through the microphone, then send the recorded voice to the voice recognition module for processing, and then obtain voice text data based on the processed text information. On the other hand, the smart glasses device can record the current scene through the camera, then send the recorded video stream to the image recognition module for processing, and then obtain scene image data based on the processed image data. Subsequently, the smart glasses device can send the voice text data and scene image data to the server based on the network communication protocol for processing by the server.

[0122] After the server processes, the analyzed voice content in text format and the replied order information in text format can be sent to the smart glasses device based on a network communication protocol. On the one hand, the smart glasses device can process the voice content in text format through a text-to-speech module to obtain voice data corresponding to the voice content, and play the voice data corresponding to the voice content through a speaker. On the other hand, the smart glasses device can process the order information in text format through a text display module to obtain image data corresponding to the order information, and display the image data corresponding to the order information through a display screen.

[0123] In some embodiments, the smart wearable device includes a hardware part and a software part. The hardware part includes a microphone, a camera, a display screen, a speaker, and a battery module. The software part includes a voice SDK, a user-friendly glasses interaction UI, multi-modal speech recognition, speech synthesis, image recognition, a perception data collection and processing module, a behavior module, a warehouse management database (recording commodity / stock information), and a memory module (large model).

[0124] Figure 7 FIG. 7 is a structural block diagram of a smart wearable device 700 for warehouse picking according to an exemplary embodiment. Referring to FIG. 7, Figure 7 The smart wearable device 700 for warehouse picking includes: A voice interaction unit 710 configured to collect multi-modal voice data associated with a picking order item of a delivery person, and output synthesized voice corresponding to guide content information related to the picking order to the delivery person through a voice playing mode. An image collection unit 720 configured to collect image data associated with the picking order item of the delivery person. An image projection unit 730 configured to output guide content information related to the picking order to the delivery person through an augmented reality projection mode, wherein the augmented reality projection mode allows the delivery person to view the guide content information while observing a real scene environment of a stock item.

[0125] The smart wearable device 700 for warehouse picking can also perform other steps of the above-mentioned method for picking a warehouse using a smart wearable device through the voice interaction unit 710, the image collection unit 720, and the image projection unit 730. Details are not described herein.

[0126] In some embodiments, the smart wearable device 700 for warehouse picking can further include a communication module configured to perform data transmission with a picking and delivery management service.

[0127] The intelligent device for warehouse picking can be an intelligent device, for example, an intelligent helmet or an intelligent glasses, wherein the intelligent glasses can be AR intelligent glasses.

[0128] Figure 8 Fig. 8 is a structural block diagram of an intelligent device for warehouse picking according to an exemplary embodiment. Referring to Fig. 8, Figure 8 The intelligent device for warehouse picking 800 comprises: an information collection module 810 configured to collect multi-modal voice data associated with a sorting order of a delivery staff and image data associated with the sorting order of the delivery staff; an information identification module 820 configured to analyze the multi-modal voice data to determine voice instructions associated with the sorting order of the delivery staff and identify the image data to determine information of goods associated with the sorting order of the delivery staff; a behavior processing module 830 configured to invoke a corresponding behavior processing agent to process according to at least one of the voice instructions associated with the sorting order of the delivery staff, the information of goods associated with the sorting order of the delivery staff, and a behavior stage of the delivery staff in the target warehouse, to obtain guidance content information related to the sorting order output to the delivery staff wearing the intelligent device; an information output module 840 configured to output the guidance content information related to the sorting order to the delivery staff through voice playback of synthesized voice or through an augmented reality projection mode, wherein the augmented reality projection mode enables the delivery staff to view the guidance content information while observing a real scene environment of goods in a goods location.

[0129] Optionally, the behavior processing module 830 is further configured to invoke a dialogue agent to identify a semantic intention of the delivery staff based on the voice instructions associated with the sorting order of the delivery staff, and process the voice instructions associated with the sorting order of the delivery staff according to the semantic intention to answer a picking question of the delivery staff about the sorting order.

[0130] Optionally, the behavior processing module 830 is further configured to, in the case of receiving an order query request, invoke an order query agent to determine sorting order information of the delivery staff in the target warehouse according to order query voice instructions of the delivery staff, and determine corresponding guidance content information according to the sorting order information of the delivery staff in the target warehouse.

[0131] Optionally, the behavior processing module 830 is further configured to, in the case that the current behavior stage is the stage of guiding the goods location, call the goods location guiding agent to determine, according to the goods information determined by the goods location image during the process of the delivery staff going to the goods location where the to-be-picked goods are located in the target warehouse, whether the delivery staff has arrived at the goods location where the to-be-picked goods are located in the target warehouse, and determine the corresponding guiding content information according to the result of whether the delivery staff has arrived at the goods location where the to-be-picked goods are located in the target warehouse.

[0132] Optionally, the behavior processing module 830 is further configured to, in the case that the current behavior stage is the stage of picking the to-be-picked goods, call the picking agent to perform goods state feature recognition on the image of the picked goods according to the image of the picked goods, determine the comparison result of the information of the picked goods and the information of the to-be-picked goods according to the goods state feature recognition result, and determine the corresponding guiding content information according to the comparison result of the information of the picked goods and the information of the to-be-picked goods.

[0133] The intelligent device for warehouse picking 800 can also perform other steps in the above-mentioned method for picking goods using an intelligent wearable device through the information acquisition module 810, the information recognition module 820, the behavior processing module 830, and the information output module 840, which will not be described here.

[0134] The AR smart glasses provided by the embodiments of the present disclosure include a frame, a microphone, a camera, a projection module, or an optical machine, a lens with an optical waveguide element, a speaker, and a battery module.

[0135] The embodiments of the present disclosure further provide a smart helmet device, which includes the AR smart glasses described above.

[0136] The preferred embodiments of the present disclosure are described in detail above with reference to the drawings, but the present disclosure is not limited to the specific details in the above-described embodiments. Within the technical concept of the present disclosure, various simple modifications can be made to the technical solutions of the present disclosure, and these simple modifications all belong to the protection scope of the present disclosure.

[0137] In addition, it should be noted that each specific technical feature described in the above specific embodiments can be combined in any appropriate manner without contradiction, and in order to avoid unnecessary repetition, the present disclosure will not describe various possible combinations again.

[0138] In addition, various different embodiments of the present disclosure can also be combined in any appropriate manner, as long as they do not deviate from the idea of the present disclosure, and they should also be considered as disclosed by the present disclosure.

Claims

1. A method for picking goods using a smart wearable device, characterized in that: The method comprises: Acquiring environmental data collected by the smart wearable device, the environmental data including voice data associated with the deliveryman's sorted order goods collected by the voice collection device of the smart wearable device, and image data associated with the deliveryman's sorted order goods collected by the image collection device of the smart wearable device; parsing the voice data to determine the voice instructions of the delivery person related to the sorting order, and recognizing the image data to determine the product information of the delivery person related to the sorting order; Obtaining guidance content information related to the sorting order output to the delivery person wearing the smart wearable device based on at least one of the voice command, the product information, and the delivery person's current picking behavior stage in the target warehouse; The guidance content information related to the sorting order is outputted through voice playback of the smart wearable device or through augmented reality projection, wherein the augmented reality projection method enables the delivery person to see the guidance content information while observing the real-life environment of the goods in the storage location.

2. The method according to claim 1, characterized in that The obtaining, based on at least one of the voice command, the product information, and the delivery person's current picking behavior stage in the target warehouse, guidance content information related to the sorting order output to the delivery person wearing the smart wearable device includes: Understanding the semantic intent of the deliveryman's voice data collected by the smart wearable device, and generating response information that corresponds to the deliveryman's voice data and is associated with the deliveryman's current sorting action based on at least one of the deliveryman's current picking behavior stage in the target warehouse, scene image information related to the deliveryman's sorting action determined by image data collected by the smart wearable device, and product information; The voice playback device of the smart wearable device outputs the answer information.

3. The method according to claim 1, characterized in that The obtaining, based on at least one of the voice command, the product information, and the delivery person's current picking behavior stage in the target warehouse, guidance content information related to the sorting order output to the delivery person wearing the smart wearable device includes: When the current behavior stage is the shelf guidance stage, determining whether the delivery person has arrived at the shelf location of the to-be-picked items in the target warehouse based on the shelf location image captured by the smart wearable device while the delivery person is heading towards the shelf location of the to-be-picked items in the target warehouse, where the to-be-picked items are items in the sorting order corresponding to the delivery person in the target warehouse; When it is determined that the delivery person wearing the smart wearable device has arrived at the cargo location where the goods to be picked are located, determining at least one of the picking voice prompt or the commodity details text information of the goods to be picked included in the guidance content information; and / or When it is determined that the delivery person wearing the smart wearable device has not arrived at the storage location where the goods to be picked are located, the picking guidance information included in the guidance content information is determined, and the picking guidance information is used to guide the delivery person to the storage location where the goods to be picked are located in the target warehouse. The picking guidance information includes a voice prompt guiding the delivery person's moving direction, or a path image guiding the delivery person's moving direction, or at least one of a display image of the goods to be picked.

4. The method according to claim 2, characterized in that The determining of the goods information based on the image of the delivery person in the process of going to the goods location in the target warehouse where the goods to be picked are located, collected by the smart wearable device, to determine whether the delivery person has arrived at the goods location in the target warehouse where the goods to be picked are located, includes: Performing location environment feature recognition on a location image of the delivery person in the process of the delivery person going to the location where the goods to be picked are located in the target warehouse, and obtaining a location environment feature recognition result, wherein the goods to be picked are the goods in the sorting order corresponding to the delivery person in the target warehouse; When it is determined based on the result of the identification of the cargo location environment characteristics that the cargo location image of the delivery person in the process of heading to the cargo location where the goods to be picked are located in the target warehouse contains an environment that is consistent with the cargo location where the goods to be picked are located, it is determined that the delivery person has arrived at the cargo location where the goods to be picked are located in the target warehouse.

5. The method according to claim 1, wherein The method further comprises: Recognize the picking instructions included in the voice data and determine that the delivery person's behavior stage enters the cargo location guidance stage; Obtaining a sorting order associated with a delivery-end account corresponding to the smart wearable device, determining a location of the items to be picked in the sorting order in the target warehouse based on a storage record of the items to be picked in the sorting order at a corresponding storage location in the target warehouse, and determining, based on a current location of a delivery person wearing the smart wearable device and the location of the items to be picked in the sorting order in the target warehouse, picking guidance information for the items to be picked in the sorting order in the target warehouse to be output to the delivery person; The picking guidance information is output to the delivery person through voice playback of the smart wearable device or through augmented reality projection.

6. The method according to claim 1, characterized in that The obtaining, based on at least one of the voice command, the product information, and the delivery person's current picking behavior stage in the target warehouse, guidance content information related to the sorting order output to the delivery person wearing the smart wearable device includes: When the current behavior stage is the stage of picking the goods to be picked, the image acquisition device of the delivery person's smart wearable device obtains an image of the picked goods, performs goods status feature recognition on the image of the picked goods, and obtains a goods status feature recognition result. The goods to be picked are the goods in the sorting order corresponding to the delivery person in the target warehouse; When it is determined that the information of the goods picked up by the delivery person matches the information of the goods to be picked up based on the result of the cargo status feature identification, it is determined that the guidance content information includes at least one of a voice prompt of completion of picking of the goods to be picked up outputted through voice playback of the smart wearable device, or a picking prompt of the next goods to be picked up outputted to the delivery person through augmented reality projection.

7. The method according to claim 6, characterized in that The method further comprises: When it is determined that the comparison result of the information of the goods picked up by the delivery person and the information of the goods to be picked up does not match based on the result of the goods status feature identification, it is determined that the guidance content information includes at least one of a voice prompt output by voice playback of the smart wearable device that the information of the picked up goods does not match the information of the goods to be picked up, or a prompt output to the delivery person by augmented reality projection that the information of the picked up goods does not match the information of the goods to be picked up.

8. The method according to claim 1, characterized in that The method further comprises: Obtain information on multiple goods included in the sorting order, determine the picking order of the multiple goods, determine information on goods to be picked based on the picking order, output the information on goods to be picked through the smart wearable device, and guide the delivery person to pick goods at the target storage location corresponding to the goods in the order of the output information on goods to be picked.

9. A smart wearable device for warehouse picking, characterized in that: include: A voice interaction unit, configured to collect multimodal voice data uttered by a delivery person and associated with the goods ordered by the delivery person, and output synthesized voice corresponding to guidance content information related to the sorting order to the delivery person through voice playback; An image acquisition unit, configured to acquire image data associated with the delivery person's sorted order items; An image projection unit is used to output guidance content information related to the sorting order to the delivery person through augmented reality projection, wherein the augmented reality projection method allows the delivery person to view the guidance content information while observing the real-life environment of the goods in the storage location.

10. An intelligent device for warehouse picking, characterized in that: include: An information collection module, configured to collect multimodal voice data associated with the deliveryman's sorted order goods, and to collect image data associated with the deliveryman's sorted order goods; an information recognition module, configured to parse the multimodal voice data to determine the delivery person's voice instructions related to the sorting order, and recognize the image data to determine the delivery person's product information related to the sorting order; a behavior processing module, configured to invoke a corresponding behavior processing agent for processing based on at least one of the delivery person's voice instruction associated with the sorting order, the delivery person's product information associated with the sorting order, and the delivery person's current picking behavior stage in the target warehouse, thereby obtaining guidance content information related to the sorting order to be output to the delivery person wearing the smart wearable device; An information output module is used to output guidance content information related to the sorting order to the delivery person through voice playback of synthesized speech or through augmented reality projection, wherein the augmented reality projection method allows the delivery person to view the guidance content information while observing the actual environment of the goods in the storage location.

Images