Article delivery method and device, storage medium and electronic equipment

Abstract

The embodiment of the invention discloses an article issuing method and device, a storage medium and electronic equipment, and the method comprises the steps: monitoring multi-mode scene information in a home scene based on an article issuing task, carrying out the recognition of a visiting event through employing an intelligent large model based on the multi-mode scene information, and carrying out the recognition of the visiting event. And obtaining an article pickup event for the pickup service object, generating a pickup interactive dialogue scheme for the dispatched article, and outputting the pickup reply content to the pickup service object based on the pickup interactive dialogue scheme. The pick-up reply content is used for indicating the pick-up service object to trigger an article sending process of the goods to be sent.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a method, device, storage medium, and electronic device for sending items on behalf of others. Background Art

[0002] With the rapid development of smart home and artificial intelligence technologies, home lifestyles are undergoing profound changes. Modern users' demand for express delivery services is gradually shifting from traditional offline processing to online and smart home services. Especially in the home environment, door-to-door delivery services have become an important way to improve user convenience. Summary of the Invention

[0003] The embodiments of this specification provide a method, device, storage medium, and electronic device for sending items on behalf of others. The technical solutions are as follows:

[0004] In a first aspect, embodiments of this specification provide a method for sending items on behalf of others, which is applied to smart home devices. The method includes:

[0005] Based on the item delivery task, monitor the multimodal scene information in the home scene;

[0006] Based on the multimodal scene information, an intelligent large model is used to identify incoming visit events, obtain item pickup events for the pickup service object, and generate an interactive dialogue solution for the pickup of the items sent on behalf of the recipient;

[0007] The pickup reply content is output to the pickup service object based on the pickup interactive dialogue solution, and the pickup reply content is used to instruct the pickup service object to trigger the item delivery process of the item delivery agent.

[0008] In a feasible implementation, the method of performing scene recognition based on the multimodal scene information using an intelligent large model to obtain an item pickup event for a pickup service object and generating an interactive dialogue solution for pickup of the item to be dispatched includes:

[0009] Based on the multimodal scene information, an intelligent large model is used to identify a visitor event for a visitor object to obtain a visitor identification result, and based on the visitor identification result, an item pickup event for a pickup service object is obtained;

[0010] The intelligent big model is used to generate dialogue content for the pickup service object to obtain a pickup interactive dialogue plan.

[0011] In a feasible implementation, the step of identifying a visitor event for a visitor object using an intelligent large model based on the multimodal scene information to obtain a visitor identification result, and obtaining an item pickup event for a pickup service object based on the visitor identification result includes:

[0012] Extracting visitor scene features corresponding to the visitor object using an intelligent large model based on the multimodal scene information, and determining visitor clothing features and visitor voice intentions of the visitor object based on the visitor scene features;

[0013] The intelligent big model verifies the pickup visit event based on the visitor's clothing features and the visitor's voice intention to obtain a visitor verification result;

[0014] An item pickup event for a pickup service object is obtained through the intelligent big model based on the visit verification result.

[0015] In a feasible implementation, the item delivery task includes a plurality of reference item delivery tasks corresponding to the items to be delivered, and obtaining an item pickup event for a pickup service object based on the visitor verification result by the intelligent big model includes:

[0016] Determine, by the intelligent big model, a reference item pickup event corresponding to each reference item dispatching task based on the visit verification result;

[0017] The service provider information and object identity information for the pickup service object are determined through the intelligent big model, and the real-time scheduling information of the delivery object corresponding to each of the reference item pickup events is obtained. Based on the service provider information, the object identity information and the real-time scheduling information of the delivery object, the item pickup event for the pickup service object is determined from each of the reference item pickup events.

[0018] In a feasible implementation, the generating of the dialogue content for the pickup service object by the intelligent big model to obtain the pickup interactive dialogue solution includes:

[0019] Determine the service provider type corresponding to the current pickup scenario through the intelligent big model, and determine the service provider's shipping order elements based on the service provider type;

[0020] Obtaining user task context information and user shipping order information corresponding to the item shipping task, and determining order element information corresponding to the service provider's shipping order element based on the user task context information and the user shipping order information;

[0021] Based on the order element information, a dialogue process template is called to generate a pickup interactive dialogue plan.

[0022] In a feasible implementation manner, outputting the pickup reply content to the pickup service object based on the pickup interactive dialogue solution includes:

[0023] Determining the dialogue behavior information of the pickup service object based on the multimodal scenario information, and determining the event task state corresponding to the item pickup event;

[0024] Based on the pickup interactive dialogue plan, the dialogue behavior information, and the event task status, determining the current task dialogue node, and obtaining the pickup reply content corresponding to the task dialogue node;

[0025] The pickup reply content corresponding to the current task dialogue node is output to the pickup service object based on the current task dialogue node.

[0026] In a feasible implementation manner, outputting the pickup reply content corresponding to the current task dialogue node to the pickup service object based on the current task dialogue node includes:

[0027] If the current task dialogue node is a pickup process node type, performing the step of determining the dialogue behavior information of the pickup service object based on the multimodal scenario information and determining the event task state corresponding to the item pickup event;

[0028] If the current task dialogue node is of the pickup completion type, then determine that the pickup service object triggers the item delivery process for the item being delivered on behalf of another, or determine that the pickup service object triggers the item delivery process for the item being delivered on behalf of another and generates task completion information for the item delivery task.

[0029] In a feasible embodiment, the method further includes:

[0030] Responding to a user's item delivery instruction for an intelligent agent service, obtaining item delivery information;

[0031] An item delivery task is generated based on the item delivery information.

[0032] In a second aspect, an embodiment of this specification provides an item delivery device, the device comprising:

[0033] The monitoring module is used to monitor multimodal scene information in home scenarios based on item delivery tasks;

[0034] An identification module, configured to identify incoming visit events using an intelligent large model based on the multimodal scene information, obtain item pickup events for pickup service recipients, and generate an interactive dialogue solution for item pickup;

[0035] The dialogue module is used to output the pickup reply content to the pickup service object based on the pickup interactive dialogue solution, and the pickup reply content is used to instruct the pickup service object to trigger the item delivery process of the item delivery.

[0036] In a feasible implementation, the method of performing scene recognition based on the multimodal scene information using an intelligent large model to obtain an item pickup event for a pickup service object and generating an interactive dialogue solution for pickup of the item to be dispatched includes:

[0037] Based on the multimodal scene information, an intelligent large model is used to identify a visitor event for a visitor object to obtain a visitor identification result, and based on the visitor identification result, an item pickup event for a pickup service object is obtained;

[0038] The intelligent big model is used to generate dialogue content for the pickup service object to obtain a pickup interactive dialogue plan.

[0039] In a feasible implementation, the step of identifying a visitor event for a visitor object using an intelligent large model based on the multimodal scene information to obtain a visitor identification result, and obtaining an item pickup event for a pickup service object based on the visitor identification result includes:

[0040] Extracting visitor scene features corresponding to the visitor object using an intelligent large model based on the multimodal scene information, and determining visitor clothing features and visitor voice intentions of the visitor object based on the visitor scene features;

[0041] The intelligent big model verifies the pickup visit event based on the visitor's clothing features and the visitor's voice intention to obtain a visitor verification result;

[0042] An item pickup event for a pickup service object is obtained through the intelligent big model based on the visit verification result.

[0043] In a feasible implementation, the item delivery task includes a plurality of reference item delivery tasks corresponding to the items to be delivered, and obtaining an item pickup event for a pickup service object based on the visitor verification result by the intelligent big model includes:

[0044] Determine, by the intelligent big model, a reference item pickup event corresponding to each reference item dispatching task based on the visit verification result;

[0045] The service provider information and object identity information for the pickup service object are determined through the intelligent big model, and the real-time scheduling information of the delivery object corresponding to each of the reference item pickup events is obtained. Based on the service provider information, the object identity information and the real-time scheduling information of the delivery object, the item pickup event for the pickup service object is determined from each of the reference item pickup events.

[0046] In a feasible implementation, the generating of the dialogue content for the pickup service object by the intelligent big model to obtain the pickup interactive dialogue solution includes:

[0047] Determine the service provider type corresponding to the current pickup scenario through the intelligent big model, and determine the service provider's shipping order elements based on the service provider type;

[0048] Obtaining user task context information and user shipping order information corresponding to the item shipping task, and determining order element information corresponding to the service provider's shipping order element based on the user task context information and the user shipping order information;

[0049] Based on the order element information, a dialogue process template is called to generate a pickup interactive dialogue plan.

[0050] In a feasible implementation manner, outputting the pickup reply content to the pickup service object based on the pickup interactive dialogue solution includes:

[0051] Determining the dialogue behavior information of the pickup service object based on the multimodal scenario information, and determining the event task state corresponding to the item pickup event;

[0052] Based on the pickup interactive dialogue plan, the dialogue behavior information, and the event task status, determining the current task dialogue node, and obtaining the pickup reply content corresponding to the task dialogue node;

[0053] The pickup reply content corresponding to the current task dialogue node is output to the pickup service object based on the current task dialogue node.

[0054] In a feasible implementation manner, outputting the pickup reply content corresponding to the current task dialogue node to the pickup service object based on the current task dialogue node includes:

[0055] If the current task dialogue node is a pickup process node type, performing the step of determining the dialogue behavior information of the pickup service object based on the multimodal scenario information and determining the event task state corresponding to the item pickup event;

[0056] If the current task dialogue node is of the pickup completion type, then determine that the pickup service object triggers the item delivery process for the item being delivered on behalf of another, or determine that the pickup service object triggers the item delivery process for the item being delivered on behalf of another and generates task completion information for the item delivery task.

[0057] In a feasible embodiment, the device is further used for:

[0058] Responding to a user's item delivery instruction for an intelligent agent service, obtaining item delivery information;

[0059] An item delivery task is generated based on the item delivery information.

[0060] In a third aspect, an embodiment of this specification provides a computer storage medium, wherein the computer storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded by a processor and executing the above-mentioned method steps.

[0061] In a fourth aspect, an embodiment of this specification provides an electronic device, which may include: a processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the above-mentioned method steps.

[0062] The beneficial effects of the technical solutions provided by some embodiments of this specification include at least:

[0063] In one or more embodiments of this specification, the smart home device monitors the multimodal scene information in the home scene based on the item delivery task, uses the intelligent big model to identify the visit event based on the multimodal scene information, obtains the item pickup event for the pickup service object, and generates the pickup interactive dialogue plan for the item delivery, and outputs the pickup reply content to the pickup service object based on the pickup interactive dialogue plan to instruct the pickup service object to trigger the item delivery process for the item delivery. A complete, closed-loop smart home item delivery processing chain is formed. Through pre-defined business rules, knowledge base mapping and multi-round dialogue mechanism, not only can the order elements be efficiently captured and confirmed, but also the user experience and process automation level can be greatly improved in actual operation, thereby realizing an efficient, convenient and safe express delivery process. This process not only effectively reduces the repeated communication between users and couriers, improves the pickup efficiency, but also greatly improves the user experience and system reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0065] Figure 1 This is a flowchart of a method for sending items on behalf of another provided in an embodiment of this specification;

[0066] Figure 2 This is a schematic diagram of an item delivery interaction between a user and an intelligent agent service provided in an embodiment of this specification;

[0067] Figure 3 This is a flow chart of an event and dialogue determination process provided by an embodiment of this specification;

[0068] Figure 4This is a flow chart of a visit time processing provided by an embodiment of this specification;

[0069] Figure 5 This is a flowchart of determining a pickup event provided by an embodiment of this specification;

[0070] Figure 6 This is a flow chart of a conversation content generation process provided by an embodiment of this specification;

[0071] Figure 7 This is a flow chart of outputting interval reply content provided by an embodiment of this specification;

[0072] Figure 8 This is a structural diagram of an item delivery device provided in an embodiment of this specification;

[0073] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this specification;

[0074] Figure 10 This is a schematic diagram of the structure of the operating system and user space provided in the embodiments of this specification;

[0075] Figure 11 yes Figure 10 The architecture diagram of the Android operating system;

[0076] Figure 12 yes Figure 10 Architecture diagram of the IOS operating system. DETAILED DESCRIPTION

[0077] The following will be combined with the drawings in the embodiments of this specification to clearly and completely describe the technical solutions in the embodiments of this specification. Obviously, the embodiments described are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this specification.

[0078] In the description of this specification, it should be understood that the terms "first", "second", etc. are used for descriptive purposes only and should not be understood as indicating or implying relative importance. In the description of this specification, it should be noted that, unless otherwise expressly specified and limited, "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices. For those of ordinary skill in the art, the specific meanings of the above terms in this specification can be understood according to the specific circumstances. In addition, in the description of this specification, unless otherwise specified, "multiple" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0079] In related technologies, with the rapid development of smart homes and logistics services, more and more users have demands for express delivery of items, and usually choose to enjoy door-to-door item collection services at home. However, for door-to-door item collection services, users often need to wait for the courier to come to their home. Due to the uncertainty of the door-to-door delivery time and the inefficiency of traditional interaction methods, long waiting times have become an important factor affecting the convenience and experience of users' lives.

[0080] The present specification is described in detail below with reference to specific embodiments.

[0081] In one embodiment, Figure 1 As shown, a method for item delivery is proposed. This method can be implemented using a computer program and can be run on an item delivery device based on the von Neumann architecture. The computer program can be integrated into an application or run as a standalone tool application. The item delivery device can be a smart home device.

[0082] This specification involves a method for sending items on behalf of others. After a user initiates a door-to-door express delivery request at home, the user does not need to wait at home for the pickup service object to collect the items to be sent. The user can complete the item delivery through smart home devices such as smart cameras, smart doorbells, and smart door locks arranged around the user's home, realizing automatic item delivery by door-to-door express delivery service, improving the user experience, and providing a new door-to-door pickup model for the express logistics industry.

[0083] Specifically, the item delivery method includes:

[0084] S102: Based on the item delivery task, monitor multimodal scene information in the home scene;

[0085] Item delivery task: refers to the task initiated by the user, which requires the assistance of smart home devices to complete the transfer or delivery of items, and the task of sending items by autonomously interacting with the pickup service object (such as a courier) on behalf of the user. In the item delivery link, the user can place the items to be sent within the monitoring range of the smart home device. When the pickup service object (such as a courier) comes to pick up the items, the smart home device replaces or assists the user in completing the transfer or delivery of items by autonomously interacting with the pickup service object (such as a courier).

[0086] Multimodal scene information: covers different data types collected by multiple sensors, such as video (images captured by cameras), audio (conversations or ambient sounds recorded by microphones), text (voice transcription from smart devices), and other environmental sensor data (such as door sensors and motion sensors).

[0087] Schematically, in the user's home scene, smart home devices (such as smart door locks, doorbell cameras, indoor sensors, etc.) collect on-site information in the home scene (such as the monitoring scene of covered objects) in real time. Smart home devices will synchronously integrate multimodal data from cameras, microphones, sensors, etc. to form multimodal scene information in the monitored home scene.

[0088] In a feasible implementation, the user inputs an item delivery instruction for the agent service to generate an item delivery task, specifically:

[0089] In response to a user's item delivery instruction for an intelligent agent service, item delivery information is acquired, and an item delivery task is generated based on the item delivery information.

[0090] Item delivery instructions: A request sent by a user through a smart terminal (such as a mobile phone app, voice assistant, or smart home device) to entrust the system to handle the delivery of an item (for example, "Please help me deliver the package").

[0091] Intelligent agent service: A human-computer interaction system built based on a large model, responsible for receiving, understanding and executing user instructions, while scheduling the back-end large model service to complete tasks.

[0092] Item delivery information: includes the user's order data (such as order number, package details, delivery address, pickup code, etc.) and logistics requirements. This information is the basis for generating item delivery tasks.

[0093] Item delivery tasks: Based on user instructions and acquired order information, specific task objects for smart home devices are automatically generated, including task numbers, processing status, key parameters, etc., for subsequent logistics operations and task tracking.

[0094] Optionally, the user can enter an "on-demand delivery" instruction for an item on the terminal device. The intelligent service front-end parses the instruction through the natural language processing (NLP) module to identify the core requirements and related information prompts in the instruction. After parsing the user's instruction, the intelligent service constructs a preliminary task context, which includes the user's identity, request type, and possible order-related prompts. Based on the task context, the database interface is called or the user's pre-entered order data is queried to obtain the item delivery information related to the item delivery instruction. If the order information is incomplete or ambiguous, the intelligent service can proactively confirm it through dialogue, such as asking: "What is the order number of the package you purchased yesterday?" to ensure the accuracy of subsequent operations.

[0095] Furthermore, the acquired order information is integrated with user instructions to generate a standardized item delivery task. This task includes detailed parameters such as user identity, order number, pickup code, address, and logistics requirements. After the task is generated, the task status is updated (such as "pending pickup" or "processing") and sent to the logistics management system, triggering subsequent automated operations (such as notifying the courier to pick up the item).

[0096] For examples, see Figure 2 , Figure 2 It is a schematic diagram of the interaction between users and intelligent agent services. Figure 2 The dialog interface between the user and the intelligent assistant (named "Xiao An") corresponding to the intelligent agent service is shown. The user enters an item delivery instruction into the intelligent assistant: they have a JD Express that needs to be shipped ("I have a JD Express that needs to be shipped today"). The user further provides a pickup code ("The pickup code is 1234") and indicates that they have placed the package on the cabinet at the door. At this point, the item delivery information (including but not limited to the pickup code) is obtained, and an item delivery task is generated based on the item delivery information.

[0097] As can be seen, users can use the smart assistant to record or convey express delivery information in order to establish an item delivery task. The item delivery task will be synchronized with the smart home device so that the courier can smoothly pick up the item through the smart home device when the courier arrives. From the conversation, it can be seen that the user only needs to place the package at the designated location at the door and provide the pickup code, and the smart assistant will communicate with the courier, reducing the user's repeated confirmation or waiting time. This process demonstrates that the item delivery method described in this manual can achieve an "unattended" delivery method in the smart home scenario, and the user can complete the delivery process at home or remotely through conversation.

[0098] S104: Using the intelligent big model to identify incoming visit events based on the multimodal scene information, obtaining an item pickup event for the pickup service object, and generating an interactive dialogue solution for pickup of the item to be sent on behalf of the recipient;

[0099] Intelligent big model: A model obtained by adapting the basic big model to the item delivery scenario;

[0100] Visitor event recognition: Use large models to perform semantic understanding and pattern matching on the collected multimodal data to identify whether there is an event of a courier visiting the door and whether the event is related to item pickup.

[0101] Item pickup event: After identifying the incoming event, confirm the pickup behavior involved in the current event, that is, determine whether the current visiting object is a courier who performs the item delivery task.

[0102] Pickup interactive dialogue solution: Based on the recognition results, an interactive question-and-answer process is generated for the pickup service object, aiming to proactively collect and confirm key order elements (such as pickup code, order number, item quantity, etc.).

[0103] Schematically, after receiving multimodal scene data through the intelligent big model, it determines whether a courier has visited the door and confirms that this event matches the item pickup task. Combined with a predefined knowledge base of service provider and order element mappings in the backend, the intelligent big model semantically decodes the visit event and extracts the order information and pickup code. Based on this extracted information and the current task status, the intelligent big model constructs an interactive Q&A template with the pickup service recipient, generating a pickup interaction plan including questions, confirmations, pickup code interaction, and exception notifications.

[0104] For example, in the multimodal data collection described above, the system captures images and spoken language (e.g., "Hello, I'm here to pick up a package"). The intelligent big model then analyzes this multimodal data and identifies the event as a package pickup. Subsequently, the intelligent big model compares the event with the order knowledge base and determines that the current pickup event likely corresponds to a user's item delivery order. Based on this, the intelligent big model generates an interactive dialogue, such as, "Hello, we've detected that your order number for pickup is XXXX. Please confirm," thus laying the foundation for subsequent confirmation.

[0105] S106: Outputting the pickup reply content to the pickup service object based on the pickup interactive dialogue solution, wherein the pickup reply content is used to instruct the pickup service object to trigger the item delivery process of the item delivery agent.

[0106] Pickup response: This is the specific response information generated by the smart home device based on the interactive dialogue solution and output to the pickup service recipient (such as the courier). This response not only contains confirmation information but also indicates subsequent actions.

[0107] Item Shipping Process: The user's items are triggered by the pickup service object and confirmed to go through the complete shipping process from pickup to final completion of logistics shipment, which usually includes steps such as confirming the order, scanning the express delivery note, and automatically generating the shipping record.

[0108] Schematically, in multiple rounds of interaction, after the intelligent big model confirms that all necessary order elements have been collected, multiple rounds of dialogue will be conducted with the pickup service object. Each round of dialogue may include pickup reply content, and the pickup reply content is used to instruct the pickup service object to trigger the item delivery process of the item; in each round of dialogue, the smart home device uses the intelligent big model to output the next pickup reply content (such as voice input from the courier) according to the pickup interactive dialogue plan and the current pickup service object.

[0109] Furthermore, the smart home device can transmit the pickup response content of the corresponding dialogue phase of the pickup interactive dialogue solution generated by the intelligent large-scale model to the pickup service recipient, and output it through voice or screen display. If the pickup service recipient responds, the smart home device collects the pickup service recipient's input dialogue information and calls the intelligent large-scale model again to generate the next round of pickup response content according to the pickup interactive dialogue solution. Once all dialogues are completed, the smart home device calls the logistics / express delivery platform interface to update the order status, completing the dispatch process.

[0110] Furthermore, if anomalies occur during multiple rounds of conversations, such as unclear identification, incorrect pickup codes, or a courier's lack of response, the smart home device can trigger the abnormal process according to pre-set policies, adjust the pickup interaction plan, and generate abnormal conversation handling steps, such as rechecking information or notifying the user. The entire conversation record and operation log can be stored in the backend for continuous optimization and traceability of the intelligent large-scale model.

[0111] The pickup response may include: identity confirmation, pickup code verification, item feature description, user requirements (such as insurance, courier company selection), etc.

[0112] The pickup reply content can be sent to the pickup service object in real time through a message push interface (such as the courier app, SMS, smart door lock display, etc.) to ensure that it is aware of the order information corresponding to the current pickup event. Usually, the pickup service object establishes a dialogue connection with the smart home device (for example, the smart home device provides a dialogue code, and the pickup service object establishes a dialogue connection by scanning the dialogue code).

[0113] The pickup reply content may be output by the smart home device using a microphone voice output;

[0114] Example of multi-round dialogue interaction:

[0115] The following dialogue is just an example. The actual wording can be flexibly generated by the big model based on the context.

[0116] The courier arrives at your door, and the smart home device's system detects the pickup event.

[0117] System: Executes the "Detect courier's visit, start pickup dialogue" step according to the pickup interactive dialogue plan.

[0118] The system outputs the first round of pickup response (to the courier): "Hello, this is the intelligent delivery assistant. Are you here to pick up [Zhang San]'s package?"

[0119] The courier types in a reply: "Yes, I'll pick up the package. What's the pickup code?"

[0120] System: Execute the "pickup code verification" step according to the pickup interactive dialogue plan;

[0121] System (calls the large model and generates a response): Generates the second round of pickup response content based on the "courier enters the response information" and the pickup interactive dialogue plan: "OK, your pickup code is

[1234] . Please confirm whether it is consistent with the information you recorded?"

[0122] The courier typed in a reply: "Yes, the pickup code 1234 is correct."

[0123] System: Execute the "item confirmation and additional requirements" link according to the pickup interactive dialogue plan;

[0124] System: Generates the third-round pickup response based on the courier's response and the pickup interactive dialogue plan: "Do you need to verify the number of packages or item characteristics? The user noted that this package is fragile, so please handle it with care."

[0125] The courier typed a reply: "OK, no problem."

[0126] System: Execute the "handover completed" link according to the pickup interactive dialogue plan;

[0127] System: Generates the fourth-round pickup response based on the courier's input and the pickup interactive dialogue plan: "Thank you for your cooperation! The package has been successfully delivered. The system will automatically update the order status. I wish you good luck on your work."

[0128] System backend: Update the order status to "Picked Up" and send a pickup notification to the user.

[0129] Record the conversation log and operation time;

[0130] In the embodiments of this specification, the smart home device monitors the multimodal scene information in the home scene based on the item delivery task, uses the intelligent big model to identify the visit event based on the multimodal scene information, obtains the item pickup event for the pickup service object, and generates the pickup interactive dialogue plan for the item delivery. Based on the pickup interactive dialogue plan, the pickup reply content is output to the pickup service object to instruct the pickup service object to trigger the item delivery process for the item delivery. A complete, closed-loop smart home item delivery processing chain is formed. Through pre-defined business rules, knowledge base mapping and multi-round dialogue mechanism, not only can the order elements be efficiently captured and confirmed, but also the user experience and process automation level can be greatly improved in actual operation, thereby realizing an efficient, convenient and safe express delivery process. This process not only effectively reduces the repeated communication between users and couriers, improves the pickup efficiency, but also greatly improves the user experience and system reliability.

[0131] See Figure 3 , Figure 3 This is a flow chart of an event and dialogue determination process proposed in this specification. Specifically, the following methods can be used to perform scene recognition based on the multimodal scene information using an intelligent large model to obtain an item pickup event for the pickup service object and generate an interactive dialogue solution for the pickup of the item being dispatched:

[0132] S202: Using the intelligent large model to identify a visitor event for a visitor object based on the multimodal scene information to obtain a visitor identification result, and obtaining an item pickup event for a pickup service object based on the visitor identification result;

[0133] Schematically, smart home devices (such as access control cameras, sensors, and voice recognition modules) capture images, voice, and environmental data at the door in real time to obtain multimodal scene information. The collected multimodal data is then passed to the intelligent big model. The intelligent big model combines pre-defined rules and training data to analyze the visitor's appearance, behavior, and voice information to identify whether there is a visitor. For example, the intelligent big model can use image recognition to determine whether the visitor is wearing a uniform courier uniform, and combine the voice content to determine keywords such as "pickup", thereby confirming whether the visitor is a pickup service object (such as a courier). The intelligent big model then obtains a "visitor identification result", which not only identifies the visitor's identity (such as courier, visitor, courier company personnel, etc.), but also outputs relevant contextual information such as time and location.

[0134] Furthermore, the intelligent big model generates an item pickup event based on the visitor identification results. Specifically, the intelligent big model automatically matches the visitor's pre-entered shipping order and logistics requirements based on the visitor identification results to determine the corresponding pickup service recipient. At this point, the intelligent big model generates an item pickup event containing the order number, pickup code, package details, and pickup recipient information for subsequent process invocation and tracking.

[0135] Example: Suppose a user has pre-entered the system with a delivery task for "Express delivery order XYZ123, pickup code 456789." When the door camera detects a visitor wearing a courier uniform and announcing "JD Express," the intelligent big model analyzes the image and voice data and outputs a visitor recognition result: "Courier has arrived." The intelligent big model then compares the order record to confirm that the visitor is the target of the pickup service and automatically generates an item pickup event (including information such as order XYZ123 and pickup code 456789), providing data support for subsequent pickup dialogues and logistics interface calls.

[0136] S204: Generate a dialogue content for the pickup service object through the intelligent big model to obtain a pickup interactive dialogue solution.

[0137] Exemplarily, the intelligent big model uses the generated item pickup event information (such as the order number, pickup code, pickup time, item description, etc.) and the current conversation state as context data, and uses preset conversation content to generate prompts. This prompt instructs the intelligent big model to dynamically construct multiple rounds of interactive conversation content using natural language generation technology. The prompts generated from the conversation content include necessary business rules, such as first confirming the courier's identity, then confirming the pickup code, then verifying the item details, and finally reminding customers of package characteristics (such as fragile items and insurance requirements).

[0138] Furthermore, the intelligent big model automatically plans the dialogue process according to the current status of the pickup event, and generates the first round, the second round, and so on until the final round of complete interactive dialogue plans for pickup. The interactive dialogue plan for pickup may include identity confirmation, pickup code confirmation, item quantity and status prompts, and special case handling (such as abnormal pickup code, item damage prompts, etc.). The generated interactive dialogue plan will be transmitted to the pickup service object in real time and output in the form of voice broadcast, text display or APP notification. During the actual dialogue process, if the pickup service object gives feedback, the system will collect the feedback information to update the dialogue context, and repeatedly call the intelligent big model to adjust the pickup interactive dialogue plan according to the dialogue context until the entire pickup process is completed.

[0139] In the embodiments of this specification, a multi-round interactive dialogue plan for the pickup service object is automatically generated through an intelligent large model, and the key information based on the pickup event is converted into natural language dialogue steps, effectively guiding the courier to perform operations such as identity confirmation, pickup code verification and item status confirmation, while supporting exception handling prompts to ensure that the entire pickup process is executed efficiently, accurately and intelligently.

[0140] In one possible implementation, see Figure 4 , Figure 4 This is a flow chart of visit time processing, specifically executing S202: based on the multimodal scene information, using the intelligent big model to perform visit event recognition for the visitor object to obtain a visit recognition result, and based on the visit recognition result, obtaining an item pickup event for the pickup service object, which can refer to the following method:

[0141] S302: extracting visitor scene features corresponding to the visitor object using an intelligent large model based on the multimodal scene information, and determining visitor clothing features and visitor voice intentions of the visitor object based on the visitor scene features;

[0142] Visitor scenario features: refers to the key features reflecting the visitor's status and behavior extracted from multimodal data after large-scale model analysis, such as clothing style, color, identifying symbols, as well as keywords, intonation, and intention in speech.

[0143] Visitor clothing characteristics: mainly refers to the visitor's clothing information, such as whether he is wearing a courier uniform, color, logo, etc. This information helps to determine the visitor's identity.

[0144] Visitor voice intent: refers to the intention or demand expressed by the visitor through voice, such as whether the visitor mentions keywords such as "pickup" or "express delivery", which reflects the purpose of performing the pickup operation.

[0145] Schematically, the collected images are analyzed by computer vision using an intelligent large model. At the same time, feature processing is performed on the voice and image signals to extract keywords and intent information from the voice. Feature processing combines visual and voice information to determine whether the visitor meets the standard attire of the courier (such as a blue uniform, a courier logo) and whether there are voice prompts such as "pick up" and "express delivery". Finally, a complete set of visitor scene feature data is determined, and then the visitor's clothing features and the visitor's voice intentions are parsed based on the visitor scene feature data.

[0146] S304: Verifying the pickup visit event based on the visitor's clothing features and the visitor's voice intention using the intelligent big model to obtain a visitor verification result;

[0147] Verification of pickup visit events: This refers to using a large model to compare and verify the extracted visitor's clothing features and voice intentions to determine whether the current visitor is indeed the courier performing the pickup task.

[0148] Visitor verification result: The result of the verification, usually "verification passed" or "verification failed", indicates whether the visitor object is confirmed to meet the standards of the pickup service object.

[0149] Schematically, the intelligent large-scale model compares the extracted visitor's attire with the predefined attire template for pickup service recipients. It also compares the visitor's voice intent to see if it contains keywords like "pick up." Based on the comparison results and the predefined threshold rules, it determines whether the visitor meets the "pick up service recipient" criteria. If the visitor's attire and voice meet expectations, the output is "Verification passed." Otherwise, the output is "Verification failed," allowing for subsequent exception handling or re-verification.

[0150] S306: Obtaining an item pickup event for the pickup service object based on the visitor verification result through the intelligent big model.

[0151] Item pickup event: refers to the pickup task record automatically generated by the intelligent large model after the visitor is confirmed to be a courier. It contains key data such as order number, pickup code, item information, pickup time, etc., which is used for subsequent logistics operations and order updates.

[0152] Illustratively, based on the visitor verification result at S304, if verification passes, this result is integrated with the user's pre-entered order information (such as order number, pickup code, item description, etc.) to generate integrated information. Using the intelligent big model, this integrated information is converted into a standardized item pickup event record. This record contains all necessary operational data and provides a basis for subsequent processes (such as notifying the courier and updating the logistics status). Finally, the generated item pickup event can be transmitted to the relevant logistics system, courier terminal, and smart home device system through the system interface, realizing automated task allocation and tracking.

[0153] In this document, multimodal scene information and an intelligent large-scale model are used to extract visitor clothing characteristics and voice intent. These characteristics are then used to verify the pickup visit event and obtain a verification result. Finally, based on the verification result, an item pickup event is generated for the pickup service recipient. The entire process automates and accurately identifies and generates pickup tasks, providing solid data support and efficient execution for subsequent logistics operations and user services.

[0154] In one possible implementation, see Figure 5 , Figure 5This is a flowchart of determining a pickup event. Considering that an item delivery task may include multiple reference item delivery tasks corresponding to respective items, S306 is specifically executed: obtaining an item pickup event for a pickup service object based on the visit verification result by the intelligent big model may refer to the following method:

[0155] S402: Determine, by the intelligent big model and based on the visitor verification result, a reference item pickup event corresponding to each reference item delivery task;

[0156] Reference item delivery tasks: In item delivery tasks, users may have multiple items to be delivered. Each item corresponds to a pre-defined reference delivery task, which includes order information, item characteristics and related logistics requirements.

[0157] Reference item pickup event: refers to the subtask record automatically generated by the system for each reference item delivery task, which is used to guide the courier to pick up the item or the smart home system to deliver the item.

[0158] Visitor verification result: The result obtained by multimodal data and intelligent big model verification in the previous step, which is used to confirm the identity of the current visitor (such as a courier) and the validity of the operation.

[0159] Schematically, using the visitor verification results, the intelligent big model identifies the relationship between the current visitor and reference item delivery tasks, and then processes each reference task one by one. For multiple reference item delivery tasks pre-entered by the user, the big model extracts the order number, pickup code, item description, and other information. Combined with the visitor verification results, the intelligent big model automatically generates a corresponding reference item pickup event record for each reference item delivery task. These records serve as the basis for subsequent task generation, identifying the specific information for each item that needs to be picked up by the courier.

[0160] Example: Suppose a user has two items for delivery: A and B. For item A, the pre-entered reference delivery task includes the order number "A001" and the pickup code "1111." For item B, the pre-entered task includes the order number "B001" and the pickup code "2222." When the courier arrives and is verified, the intelligent large model uses step S402 to generate a reference item pickup event for item A (recording information such as order number "A001" and pickup code "1111") and a reference item pickup event for item B (recording order number "B001" and pickup code "2222").

[0161] S404: Determine the service provider information and object identity information for the pickup service object through the intelligent big model, obtain the real-time scheduling information of the delivery object corresponding to each of the reference item pickup events, and determine the item pickup event for the pickup service object from each of the reference item pickup events based on the service provider information, the object identity information and the real-time scheduling information of the delivery object.

[0162] Service provider information: refers to the information related to the courier service provider to which the current pickup service belongs, such as the courier company name, delivery rules, logistics system interface, etc.

[0163] Object identity information: refers to the relevant data after the identity of the pickup service object (such as a courier), such as work permit information, task assignment records, etc.

[0164] Real-time dispatch information for delivery objects: real-time logistics dispatch data related to the reference item pickup event, including the delivery vehicle, the courier's current task status, and the dispatch center's allocation status.

[0165] Item pickup event: The finalized, complete pickup task record for the pickup service object, which integrates reference item pickup events and real-time scheduling information to guide the courier's actual pickup operation.

[0166] Schematically, the intelligent big model first verifies the identity of the current pickup service recipient and obtains detailed information about their courier provider, typically from an enterprise database or third-party logistics system. The system then calls the logistics platform interface to query the dispatch status of the delivery recipient corresponding to each reference item pickup event in real time, including the current vehicle schedule, courier workload, and location. Combining this information with the service provider, the recipient's identity, and the delivery recipient's real-time dispatch information, the intelligent big model further evaluates each reference item pickup event, comparing and screening all candidate reference item pickup events to determine which record best matches the current pickup service recipient. The system then integrates the item pickup events that match the pickup service recipient from multiple reference events into a final item pickup event. The system then outputs the filtered item pickup event record, which contains all necessary information (order number, pickup code, service provider, and dispatch data), serving as the basis for subsequent logistics operations and task tracking.

[0167] For example, take the previously generated reference item pickup event as an example:

[0168] The user has two candidate events (item A and item B). When the courier arrives, the system verifies the courier's identity and company using the large model, while also querying real-time dispatch data (such as task load and current location). Assuming that the real-time data indicates that the courier has been assigned task A (order "A001") in the system and has a higher match with their identity, step S404 will ultimately determine that the item pickup event is for order "A001" and ignore the candidate record for order "B001."

[0169] In this document, candidate reference item pickup events are generated for each of a user's multiple reference item delivery tasks. An intelligent large-scale model leverages service provider information, the identity of the pickup recipient, and real-time scheduling data to match and determine the unique pickup event corresponding to the current visitor from these candidate events. This ensures that the system can accurately identify the specific pickup task that the current pickup recipient should perform, improving the accuracy and intelligence of the entire delivery process.

[0170] Optional, see Figure 6 , Figure 6 This is a flow chart of conversation content generation. The following methods can be used to generate conversation content for the pickup service object using the intelligent big model to obtain a pickup interactive conversation solution:

[0171] S502: Determine the service provider type corresponding to the current pickup scenario through the intelligent big model, and determine the service provider's shipping order elements based on the service provider type;

[0172] Service Provider Type: refers to the courier or logistics service provider involved in the current pickup scenario.

[0173] Service provider shipping order elements: refers to the key information fields required by each service provider when generating a shipping order, such as order number, pickup code, item description, weight, insurance information, shipping address, etc. These elements vary between different service providers. Only by determining the information corresponding to the service provider's shipping order elements can the pickup service object trigger the item shipping process for the item.

[0174] Schematically, a large intelligent model is used to automatically identify the characteristics of the current pickup scenario through multimodal data (images, voice, sensor data, etc.) and determine the type of service provider. For example, image recognition can be used to identify keywords in the courier's uniform, license plate, or voice to distinguish different service providers. Once the service provider type is determined, the system retrieves the pre-built service provider knowledge base and extracts the corresponding shipping order elements (such as required fields and special requirements) to prepare for the subsequent generation of specific order data.

[0175] S504: Obtain user task context information and user shipping order information corresponding to the item shipping task, and determine order element information corresponding to the service provider's shipping order element based on the user task context information and the user shipping order information;

[0176] User task context information: refers to the various background data provided by the user when initiating an item delivery task, including item description, delivery requirements, time, address, contact information, etc.

[0177] User shipping order information: refers to the specific order data related to shipping entered by the user on the platform, such as order number, reserved pickup code, quantity of goods and special instructions.

[0178] Order element information: This refers to the specific values or text descriptions of the service provider's order elements in the current task, which are the parameters necessary to complete order generation.

[0179] Schematically, the control intelligent big model extracts task context information and user shipping order information from the system's recorded user-submitted item shipping tasks. The intelligent big model then matches this collected information with the service provider's shipping order elements in S502 to determine the order element information corresponding to the service provider's shipping order elements. For example, if the service provider's shipping order element requires a "pickup code," the system extracts the corresponding value from the user's order; if "item weight" is required, the weight data is parsed from the task description. The extracted and matched order element information is then integrated into a structured data object, providing complete information support for the next step in generating the dialogue process.

[0180] Example

[0181] For example, a user's delivery task contains two key pieces of data: the order number "XYZ123" and the pickup code "456789," as well as the item description "fragile, weighs 1.5 kg." For a specific express order template, the system maps this information to the "order number," "pickup code," and "item characteristics" fields, generating a complete order element information object.

[0182] S506: Based on the order element information, a dialogue process template is called to generate a pickup interactive dialogue solution.

[0183] Dialogue process template: A pre-designed interactive script template that includes various nodes of multiple rounds of dialogue (such as identity confirmation, order information verification, exception prompts, etc.). The variables in the template can be dynamically filled with order element information.

[0184] Pickup interactive dialogue plan: The large model ultimately generates a complete and adaptive dialogue plan for the current task situation, which is used to guide the courier to complete the pickup operation.

[0185] Schematically, the intelligent big model uses the order element information obtained in S504 as parameters and feeds it into a pre-set dialogue process template. Based on the business process logic in the template, the big model fills in the corresponding information, such as the order number, pickup code, and item description. It also generates a natural, coherent multi-round dialogue to create a pickup interaction plan. For example, the first round might ask for order confirmation, the second round might confirm the pickup code, and the third round might prompt special precautions. The generated pickup interaction plan serves as the current output, guiding the intelligent security device to interact with the pickup service recipient to execute the pickup operation according to the process.

[0186] In this manual, an intelligent big model is used to determine the service provider type corresponding to the current pickup scenario from multimodal scenario data, and output the key elements required for the service provider to send the order; extract the user task context and shipping order information, and map out the specific order element information; finally, by calling the preset dialogue process template, the order element information is dynamically filled in to generate a complete pickup interactive dialogue plan, thereby providing the pickup service object with accurate, real-time and personalized operation guidance, ensuring that the entire item delivery task process is executed efficiently and accurately.

[0187] Optional, see Figure 7 , Figure 7 This is a flow chart of outputting interval reply content. The following method can be used to output the pickup reply content to the pickup service object based on the pickup interactive dialogue solution:

[0188] S602: Determine the dialogue behavior information of the pickup service object based on the multimodal scenario information, and determine the event task state corresponding to the item pickup event;

[0189] Conversational behavior information of the pickup service recipient: refers to the analysis of behavioral data such as language, speaking speed, emotions, reaction speed, etc. input by the visitor (such as the courier) on-site or through the terminal to determine their current conversation status and response behavior.

[0190] Event Task Status: describes the current task process status of the item pickup event, such as "Waiting for Confirmation", "Pickup Code Verification", "Confirmed", etc.

[0191] Schematically, the system uses multimodal devices to collect images, voice, and other data from the pickup site. It then uses an intelligent big model to analyze the visitor's conversational behavior (such as voice content, intonation, and response time). The big model parses the visitor's verbal and non-verbal behavior, extracting key information to determine the status, thereby determining whether the pickup service recipient has responded, whether there are any questions, or if there are any anomalies. The intelligent big model combines the extracted conversational behavior information to determine the task status of the current item pickup event. For example, if the courier quickly and clearly confirms the pickup code, the status is "Confirmed"; if the response is slow or the answer is vague, the status may be "Pending Confirmation" or "Anomaly."

[0192] S604: Based on the pickup interactive dialogue plan, the dialogue behavior information, and the event task status, determine the current task dialogue node, and obtain the pickup reply content corresponding to the task dialogue node;

[0193] The pickup interactive dialogue solution is a pre-designed multi-round dialogue template that includes nodes such as identity confirmation, pickup code verification, item status prompts, and exception handling.

[0194] Current task dialogue node: refers to the specific step or stage currently in the entire pickup interaction process, such as "Confirm pickup code" or "Check item status".

[0195] Pickup reply content: For the current task dialogue node, the large model generates a specific reply text based on the preset template to guide the courier to take the next step.

[0196] Illustratively, the system combines the conversation behavior information and event task status obtained in S602 with the pre-generated pickup interactive dialogue plan to form a complete context. Based on the current task status (e.g., "Confirmed" or "To Be Confirmed") and user behavior data, the intelligent large-scale model automatically determines the node in the current conversation process within the pickup interactive dialogue plan. For example, if the status is "Confirmed," the current node may enter the "Follow-up Prompt" phase. Furthermore, based on the matched task dialogue node, the corresponding pickup response content can be extracted or generated from the pickup interactive dialogue plan to ensure that the language logic is consistent with business requirements.

[0197] S606: Outputting the pickup reply content corresponding to the current task dialogue node to the pickup service object based on the current task dialogue node.

[0198] In principle, the reply content can be conveyed to the pickup service recipient in real time through message push, voice broadcast, or display on the terminal device. At the same time, feedback monitoring is triggered: the system continuously monitors the feedback of the pickup service recipient. If there is no response or anomaly, the conversation status is updated in a timely manner and a subsequent reply is generated, forming a closed-loop interaction.

[0199] In this manual: multimodal data is used to extract the dialogue behavior information of the pickup service object and determine the current task status of the item pickup event; and based on the pickup interactive dialogue plan, dialogue behavior information and event task status, the current task dialogue node is automatically matched and the corresponding pickup reply content is obtained; and finally the generated reply content is output to the pickup service object in real time, ensuring that the entire dialogue process dynamically and accurately guides the courier to perform operations, thereby realizing intelligent, continuous and efficient interactive output in the item delivery task.

[0200] In a feasible implementation manner, the outputting of the pickup reply content corresponding to the current task dialogue node to the pickup service object based on the current task dialogue node may be performed in the following manner:

[0201] 1. If the current task dialogue node is a pickup process node type, then performing the step of determining the dialogue behavior information of the pickup service object based on the multimodal scenario information and determining the event task state corresponding to the item pickup event, i.e., S602;

[0202] Pickup process node type: refers to the node in the entire pickup interaction process that is currently in the operation confirmation, information verification, status update, etc. stage. At this time, the pickup process is still in the unfinished stage.

[0203] 2. If the current task dialogue node is of the pickup completion type, determine that the pickup service object triggers the item delivery process for the item being delivered on behalf of another, or determine that the pickup service object triggers the item delivery process for the item being delivered on behalf of another and generates task completion information for the item delivery task.

[0204] Pickup completion type: refers to the final node of the pickup process, at which point all key information has been confirmed to be correct, the pickup operation is complete, and the subsequent item shipping process will be triggered or task completion information will be generated.

[0205] Item delivery process: refers to the automated process from pickup to delivery in express delivery tasks, including order status updates, logistics system calls, etc.

[0206] Task completion information: records and provides feedback on the detailed information on the completion of the entire item delivery task, which will be used for subsequent task archiving, statistics and user notifications.

[0207] Specifically, when the current task dialogue node is of the pickup completion type, the system confirms that the current process has entered the end stage and all necessary pickup confirmation information has been obtained.

[0208] The system further determines and triggers subsequent operations:

[0209] One way is to directly determine the pickup service object to trigger the item delivery process, that is, the system calls the logistics interface to start the delivery operation;

[0210] Another way is to generate corresponding task completion information while triggering the item delivery process, recording all key data and operation results of this delivery task.

[0211] Finally, the system will output the completed status and operation feedback to the pickup service recipient in the form of a message or voice, prompting "Pickup completed" and informing the subsequent logistics delivery arrangements or task archiving status.

[0212] Example

[0213] For example, if the current task dialog node is set to "Pickup Completed," the system confirms that the courier has completed all verification operations, triggering the item delivery process. The system's output response might be: "Pickup completed, package information confirmed, item delivery process initiated. Please pay attention to subsequent logistics arrangements." At the same time, the system generates task completion information, recording data such as the order, pickup time, and confirmation status for backend archiving and user query.

[0214] In this implementation, different processing methods are used according to the type of the current task dialogue node:

[0215] If the node belongs to the pickup process node type, the system determines the dialogue behavior information of the pickup service object through multimodal scene information, updates the task status of the item pickup event, and then outputs the corresponding process guidance reply content;

[0216] If the node is of the pickup completion type, the system determines the pickup service object to trigger the item delivery process, or generates task completion information while triggering the process, and outputs the reply content of the final operation confirmation.

[0217] This hierarchical judgment and dynamic response mechanism ensures continuity in the entire pickup interaction process while providing precise guidance at different stages, thus improving the automation and execution efficiency of express delivery tasks.

[0218] The following will be combined Figure 8 , the article distribution device provided in the embodiment of this specification is introduced in detail. It should be noted that, Figure 8 The item delivery device shown is used to execute this instruction Figures 1 to 7 For the convenience of explanation, only the part related to the embodiment of this specification is shown. For the specific technical details not disclosed, please refer to this specification. Figures 1 to 7 The embodiment shown.

[0219] See Figure 8, which shows a schematic diagram of the structure of the item delivery device according to an embodiment of this specification. The item delivery device 1 can be implemented as all or part of the device through software, hardware, or a combination of both. According to some embodiments, the item delivery device 1 includes an item delivery module 11, an item delivery module 12, and an item delivery module 13, which are specifically used to:

[0220] The monitoring module is used to monitor multimodal scene information in home scenarios based on item delivery tasks;

[0221] An identification module, configured to identify incoming visit events using an intelligent large model based on the multimodal scene information, obtain item pickup events for pickup service recipients, and generate an interactive dialogue solution for item pickup;

[0222] The dialogue module is used to output the pickup reply content to the pickup service object based on the pickup interactive dialogue solution, and the pickup reply content is used to instruct the pickup service object to trigger the item delivery process of the item delivery.

[0223] In a feasible implementation, the method of performing scene recognition based on the multimodal scene information using an intelligent large model to obtain an item pickup event for a pickup service object and generating an interactive dialogue solution for pickup of the item to be dispatched includes:

[0224] Based on the multimodal scene information, an intelligent large model is used to identify a visitor event for a visitor object to obtain a visitor identification result, and based on the visitor identification result, an item pickup event for a pickup service object is obtained;

[0225] The intelligent big model is used to generate dialogue content for the pickup service object to obtain a pickup interactive dialogue plan.

[0226] In a feasible implementation, the step of identifying a visitor event for a visitor object using an intelligent large model based on the multimodal scene information to obtain a visitor identification result, and obtaining an item pickup event for a pickup service object based on the visitor identification result includes:

[0227] Extracting visitor scene features corresponding to the visitor object using an intelligent large model based on the multimodal scene information, and determining visitor clothing features and visitor voice intentions of the visitor object based on the visitor scene features;

[0228] The intelligent big model verifies the pickup visit event based on the visitor's clothing features and the visitor's voice intention to obtain a visitor verification result;

[0229] An item pickup event for a pickup service object is obtained through the intelligent big model based on the visit verification result.

[0230] In a feasible implementation, the item delivery task includes a plurality of reference item delivery tasks corresponding to the items to be delivered, and obtaining an item pickup event for a pickup service object based on the visitor verification result by the intelligent big model includes:

[0231] Determine, by the intelligent big model, a reference item pickup event corresponding to each reference item dispatching task based on the visit verification result;

[0232] The service provider information and object identity information for the pickup service object are determined through the intelligent big model, and the real-time scheduling information of the delivery object corresponding to each of the reference item pickup events is obtained. Based on the service provider information, the object identity information and the real-time scheduling information of the delivery object, the item pickup event for the pickup service object is determined from each of the reference item pickup events.

[0233] In a feasible implementation, the generating of the dialogue content for the pickup service object by the intelligent big model to obtain the pickup interactive dialogue solution includes:

[0234] Determine the service provider type corresponding to the current pickup scenario through the intelligent big model, and determine the service provider's shipping order elements based on the service provider type;

[0235] Obtaining user task context information and user shipping order information corresponding to the item shipping task, and determining order element information corresponding to the service provider's shipping order element based on the user task context information and the user shipping order information;

[0236] Based on the order element information, a dialogue process template is called to generate a pickup interactive dialogue plan.

[0237] In a feasible implementation manner, outputting the pickup reply content to the pickup service object based on the pickup interactive dialogue solution includes:

[0238] Determining the dialogue behavior information of the pickup service object based on the multimodal scenario information, and determining the event task state corresponding to the item pickup event;

[0239] Based on the pickup interactive dialogue plan, the dialogue behavior information, and the event task status, determining the current task dialogue node, and obtaining the pickup reply content corresponding to the task dialogue node;

[0240] The pickup reply content corresponding to the current task dialogue node is output to the pickup service object based on the current task dialogue node.

[0241] In a feasible implementation manner, outputting the pickup reply content corresponding to the current task dialogue node to the pickup service object based on the current task dialogue node includes:

[0242] If the current task dialogue node is a pickup process node type, performing the step of determining the dialogue behavior information of the pickup service object based on the multimodal scenario information and determining the event task state corresponding to the item pickup event;

[0243] If the current task dialogue node is of the pickup completion type, then determine that the pickup service object triggers the item delivery process for the item being delivered on behalf of another, or determine that the pickup service object triggers the item delivery process for the item being delivered on behalf of another and generates task completion information for the item delivery task.

[0244] It should be noted that the item delivery device provided in the above embodiment only uses the division of the above functional modules as an example when executing the item delivery method. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the item delivery device provided in the above embodiment and the item delivery method embodiment are of the same concept. The implementation process is detailed in the method embodiment and will not be repeated here.

[0245] The serial numbers of the embodiments in this specification are for description only and do not represent the advantages or disadvantages of the embodiments.

[0246] The embodiment of this specification also provides a computer storage medium, which can store multiple instructions, and the instructions are suitable for being loaded and executed by a processor as described above. Figures 1 to 7 The specific implementation process of the item delivery method in the embodiment shown can be found in Figures 1 to 7 The detailed description of the illustrated embodiment will not be repeated here.

[0247] This specification also provides a computer program product, which stores at least one instruction, and the at least one instruction is loaded and executed by the processor as described above. Figures 1 to 7 The specific implementation process of the item delivery method in the embodiment shown can be found in Figures 1 to 7 The detailed description of the illustrated embodiment will not be repeated here.

[0248] Please refer to Figure 9, which shows a block diagram of the structure of an electronic device provided by an exemplary embodiment of this specification. The electronic device described in this specification may include one or more of the following components: a processor 110, a memory 120, an input device 130, an output device 140, and a bus 150. The processor 110, the memory 120, the input device 130, and the output device 140 may be connected via the bus 150.

[0249] The processor 110 may include one or more processing cores. The processor 110 utilizes various interfaces and circuits to connect various components within the electronic device. It executes instructions, programs, code sets, or instruction sets stored in the memory 120, as well as accesses data stored in the memory 120, to perform various functions of the electronic device and process data. Optionally, the processor 110 may be implemented using at least one of the following hardware forms: a digital signal processing (DSP), a field-programmable gate array (FPGA), or a programmable logic array (PLA). The processor 110 may integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. The CPU primarily handles the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem handles wireless communications. It is understood that the modem may not be integrated into the processor 110 and may be implemented separately via a communications chip.

[0250] The memory 120 may include a random access memory (RAM) or a read-only memory (ROM). Optionally, the memory 120 includes a non-transitory computer-readable storage medium. The memory 120 may be used to store instructions, programs, codes, code sets, or instruction sets. The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The operating system may be an Android system, including a system deeply developed based on the Android system, an IOS system developed by Apple, including a system deeply developed based on the IOS system or other systems. The data storage area may also store data created by the electronic device during use, such as a phone book, audio and video data, chat record data, etc.

[0251] See also Figure 10 As shown, the memory 120 can be divided into operating system space and user space. The operating system runs in the operating system space, and native and third-party applications run in the user space. In order to ensure that different third-party applications can achieve better operating results, the operating system allocates corresponding system resources to different third-party applications. However, the requirements for system resources in different application scenarios in the same third-party application are also different. For example, in the local resource loading scenario, the third-party application has higher requirements for disk reading speed; in the animation rendering scenario, the third-party application has higher requirements for GPU performance. The operating system and the third-party application are independent of each other, and the operating system often cannot perceive the current application scenario of the third-party application in a timely manner, resulting in the operating system being unable to perform targeted system resource adaptation according to the specific application scenario of the third-party application.

[0252] In order for the operating system to distinguish the specific application scenarios of third-party applications, it is necessary to open up data communication between third-party applications and the operating system so that the operating system can obtain the current scenario information of third-party applications at any time, and then perform targeted system resource adaptation based on the current scenario.

[0253] Taking the Android operating system as an example, the programs and data stored in the memory 120 are as follows: Figure 11As shown, the memory 120 may store a Linux kernel layer 320, a system runtime library layer 340, an application framework layer 360, and an application layer 380. The Linux kernel layer 320, the system runtime library layer 340, and the application framework layer 360 belong to the operating system space, and the application layer 380 belongs to the user space. The Linux kernel layer 320 provides underlying drivers for various hardware components of electronic devices, such as display drivers, audio drivers, camera drivers, Bluetooth drivers, Wi-Fi drivers, power management, etc. The system runtime library layer 340 provides major feature support for the Android system through some C / C++ libraries. For example, the SQLite library provides database support, the OpenGL / ES library provides 3D drawing support, and the Webkit library provides browser kernel support. The system runtime library layer 340 also provides the Android runtime library (Android runtime), which mainly provides some core libraries that allow developers to write Android applications using the Java language. The application framework layer 360 provides various APIs that may be used when building applications. Developers can also use these APIs to build their own applications, such as activity management, window management, view management, notification management, content provider management, package management, call management, resource management, and location management. The application layer 380 runs at least one application. These applications can be native applications that come with the operating system, such as contacts, SMS, clock, and camera applications, or third-party applications developed by third-party developers, such as games, instant messaging programs, and photo enhancement programs.

[0254] Taking the operating system as the IOS system as an example, the programs and data stored in the memory 120 are as follows: Figure 12As shown, the IOS system includes: a core operating system layer 420 (Core OS layer), a core service layer 440 (Core Services layer), a media layer 460 (Media layer), and a touchable layer 480 (Cocoa Touch Layer). The core operating system layer 420 includes the operating system kernel, drivers, and underlying program frameworks. These underlying program frameworks provide functions closer to the hardware for use by the program framework located in the core service layer 440. The core service layer 440 provides system services and / or program frameworks required by applications, such as the foundation framework, account framework, advertising framework, data storage framework, network connection framework, geographic location framework, motion framework, etc. The media layer 460 provides applications with audio-visual interfaces, such as graphics and image-related interfaces, audio technology-related interfaces, video technology-related interfaces, and wireless playback (AirPlay) interfaces for audio and video transmission technologies. The touchable layer 480 provides various commonly used interface-related frameworks for application development. The touchable layer 480 is responsible for user touch interaction operations on electronic devices. For example, local notification service, remote push service, advertising framework, game tool framework, message user interface (UI) framework, user interface UIKit framework, map framework, etc.

[0255] exist Figure 12 Among the frameworks shown, those relevant to most applications include, but are not limited to, the Foundation framework in the core services layer 440 and the UIKit framework in the touchable layer 480. The Foundation framework provides many basic object classes and data types, offering fundamental system services for all applications and having nothing to do with the UI. The classes provided by the UIKit framework are the foundational UI class library for creating touch-based user interfaces. iOS applications can use the UIKit framework to provide their UIs, providing the application infrastructure for building user interfaces, drawing, handling user interaction events, responding to gestures, and so on.

[0256] Among them, the method and principle of implementing data communication between third-party applications and the operating system in the IOS system can be referred to the Android system, and this manual will not go into details here.

[0257] Among them, the input device 130 is used to receive input instructions or data, and the input device 130 includes but is not limited to a keyboard, a mouse, a camera, a microphone or a touch device. The output device 140 is used to output instructions or data, and the output device 140 includes but is not limited to a display device and a speaker. In one example, the input device 130 and the output device 140 can be combined, and the input device 130 and the output device 140 are touch screen displays, which are used to receive touch operations on or near the user using any suitable objects such as fingers and touch pens, and to display the user interface of each application. The touch screen display is usually provided on the front panel of the electronic device. The touch screen display can be designed as a full screen, a curved screen or a special-shaped screen. The touch screen display can also be designed as a combination of a full screen and a curved screen, or a combination of a special-shaped screen and a curved screen, which is not limited in the embodiments of this specification.

[0258] In addition, those skilled in the art will understand that the structures of the electronic devices shown in the above figures do not limit the electronic devices. The electronic devices may include more or fewer components than shown, or may combine certain components, or arrange the components differently. For example, the electronic devices may also include radio frequency circuits, input units, sensors, audio circuits, wireless fidelity (WiFi) modules, power supplies, Bluetooth modules, and other components, which are not described in detail here.

[0259] In the embodiments of this specification, the execution entity of each step can be the electronic device described above. Optionally, the execution entity of each step is the operating system of the electronic device. The operating system can be Android, iOS, or other operating systems, and this embodiment of this specification does not limit this.

[0260] The electronic device of the embodiment of this specification may further be equipped with a display device, and the display device may be any device capable of realizing a display function, such as a cathode ray tube display (CR), a light-emitting diode display (LED), an electronic ink screen, a liquid crystal display (LCD), a plasma display panel (PDP), etc. The user may use the display device on the electronic device to view displayed text, images, videos and other information. The electronic device may be a smart phone, a tablet computer, a gaming device, an AR (Augmented Reality) device, a car, a data storage device, an audio playback device, a video playback device, a notebook, a desktop computing device, a wearable device such as an electronic watch, electronic glasses, an electronic helmet, an electronic bracelet, an electronic necklace, electronic clothing and the like.

[0261] exist Figure 9 In the electronic device shown, the processor 110 may be configured to call an application stored in the memory 120 and specifically perform the following operations:

[0262] Based on the item delivery task, monitor the multimodal scene information in the home scene;

[0263] Based on the multimodal scene information, an intelligent large model is used to identify incoming visit events, obtain item pickup events for the pickup service object, and generate an interactive dialogue solution for the pickup of the items sent on behalf of the recipient;

[0264] The pickup reply content is output to the pickup service object based on the pickup interactive dialogue solution, and the pickup reply content is used to instruct the pickup service object to trigger the item delivery process of the item delivery agent.

[0265] In a feasible implementation, the method of performing scene recognition based on the multimodal scene information using an intelligent large model to obtain an item pickup event for a pickup service object and generating an interactive dialogue solution for pickup of the item to be dispatched includes:

[0266] Based on the multimodal scene information, an intelligent large model is used to identify a visitor event for a visitor object to obtain a visitor identification result, and based on the visitor identification result, an item pickup event for a pickup service object is obtained;

[0267] The intelligent big model is used to generate dialogue content for the pickup service object to obtain a pickup interactive dialogue plan.

[0268] In a feasible implementation, the step of identifying a visitor event for a visitor object using an intelligent large model based on the multimodal scene information to obtain a visitor identification result, and obtaining an item pickup event for a pickup service object based on the visitor identification result includes:

[0269] Extracting visitor scene features corresponding to the visitor object using an intelligent large model based on the multimodal scene information, and determining visitor clothing features and visitor voice intentions of the visitor object based on the visitor scene features;

[0270] The intelligent big model verifies the pickup visit event based on the visitor's clothing features and the visitor's voice intention to obtain a visitor verification result;

[0271] An item pickup event for a pickup service object is obtained through the intelligent big model based on the visit verification result.

[0272] In a feasible implementation, the item delivery task includes a plurality of reference item delivery tasks corresponding to the items to be delivered, and obtaining an item pickup event for a pickup service object based on the visitor verification result by the intelligent big model includes:

[0273] Determine, by the intelligent big model, a reference item pickup event corresponding to each reference item dispatching task based on the visitor verification result;

[0274] The service provider information and object identity information for the pickup service object are determined through the intelligent big model, and the real-time scheduling information of the delivery object corresponding to each of the reference item pickup events is obtained. Based on the service provider information, the object identity information and the real-time scheduling information of the delivery object, the item pickup event for the pickup service object is determined from each of the reference item pickup events.

[0275] In a feasible implementation, the generating of the dialogue content for the pickup service object by the intelligent big model to obtain the pickup interactive dialogue solution includes:

[0276] Determine the service provider type corresponding to the current pickup scenario through the intelligent big model, and determine the service provider's shipping order elements based on the service provider type;

[0277] Obtaining user task context information and user shipping order information corresponding to the item shipping task, and determining order element information corresponding to the service provider's shipping order element based on the user task context information and the user shipping order information;

[0278] Based on the order element information, a dialogue process template is called to generate a pickup interactive dialogue plan.

[0279] In a feasible implementation manner, outputting the pickup reply content to the pickup service object based on the pickup interactive dialogue solution includes:

[0280] Determining the dialogue behavior information of the pickup service object based on the multimodal scenario information, and determining the event task state corresponding to the item pickup event;

[0281] Based on the pickup interactive dialogue plan, the dialogue behavior information, and the event task status, determining the current task dialogue node, and obtaining the pickup reply content corresponding to the task dialogue node;

[0282] The pickup reply content corresponding to the current task dialogue node is output to the pickup service object based on the current task dialogue node.

[0283] In a feasible implementation manner, outputting the pickup reply content corresponding to the current task dialogue node to the pickup service object based on the current task dialogue node includes:

[0284] If the current task dialogue node is a pickup process node type, performing the step of determining the dialogue behavior information of the pickup service object based on the multimodal scenario information and determining the event task state corresponding to the item pickup event;

[0285] If the current task dialogue node is of the pickup completion type, then determine that the pickup service object triggers the item delivery process for the item being delivered on behalf of another, or determine that the pickup service object triggers the item delivery process for the item being delivered on behalf of another and generates task completion information for the item delivery task.

[0286] In a feasible embodiment, the method further includes:

[0287] Responding to a user's item delivery instruction for an intelligent agent service, obtaining item delivery information;

[0288] An item delivery task is generated based on the item delivery information.

[0289] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware through a computer program. The program can be stored in a computer-readable storage medium, and when executed, the program can include the processes in the above-described method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory, or a random access memory.

[0290] The above disclosure is only a preferred embodiment of this specification, and certainly cannot be used to limit the scope of rights of this specification. Therefore, equivalent changes made according to the claims of this specification are still within the scope covered by this specification.

Claims

1. A method for sending goods on behalf of others, characterized in that: Applied to smart home devices, the method includes: Based on the item delivery task, monitor the multimodal scene information in the home scene; Based on the multimodal scene information, an intelligent large model is used to identify incoming visit events, obtain item pickup events for the pickup service object, and generate an interactive dialogue solution for the pickup of the items sent on behalf of the recipient; The pickup reply content is output to the pickup service object based on the pickup interactive dialogue solution, and the pickup reply content is used to instruct the pickup service object to trigger the item delivery process of the item delivery agent.

2. The method according to claim 1, characterized in that The method of using a large intelligent model to perform scene recognition based on the multimodal scene information, obtaining an item pickup event for a pickup service object, and generating an interactive dialogue solution for pickup of items on behalf of the delivery service object includes: Based on the multimodal scene information, an intelligent large model is used to identify a visitor event for a visitor object to obtain a visitor identification result, and based on the visitor identification result, an item pickup event for a pickup service object is obtained; The intelligent big model is used to generate dialogue content for the pickup service object to obtain a pickup interactive dialogue plan.

3. The method according to claim 2, characterized in that The method of using the intelligent large model to perform visit event recognition for the visitor object based on the multimodal scene information to obtain a visit recognition result, and obtaining an item pickup event for the pickup service object based on the visit recognition result includes: Extracting visitor scene features corresponding to the visitor object using an intelligent large model based on the multimodal scene information, and determining visitor clothing features and visitor voice intentions of the visitor object based on the visitor scene features; The intelligent big model verifies the pickup visit event based on the visitor's clothing features and the visitor's voice intention to obtain a visitor verification result; An item pickup event for a pickup service object is obtained through the intelligent big model based on the visit verification result.

4. The method according to claim 3, characterized in that The item delivery task includes a plurality of reference item delivery tasks corresponding to the items to be delivered, and the item pickup event for the pickup service object is obtained based on the visit verification result by the intelligent big model, including: Determine, by the intelligent big model, a reference item pickup event corresponding to each reference item dispatching task based on the visit verification result; The service provider information and object identity information for the pickup service object are determined through the intelligent big model, and the real-time scheduling information of the delivery object corresponding to each of the reference item pickup events is obtained. Based on the service provider information, the object identity information and the real-time scheduling information of the delivery object, the item pickup event for the pickup service object is determined from each of the reference item pickup events.

5. The method according to claim 2, characterized in that The generating of the interactive dialogue solution for the pickup service object by using the intelligent big model includes: Determine the service provider type corresponding to the current pickup scenario through the intelligent big model, and determine the service provider's shipping order elements based on the service provider type; Obtaining user task context information and user shipping order information corresponding to the item shipping task, and determining order element information corresponding to the service provider's shipping order element based on the user task context information and the user shipping order information; Based on the order element information, a dialogue process template is called to generate a pickup interactive dialogue plan.

6. The method according to claim 1, characterized in that The outputting the pickup reply content to the pickup service object based on the pickup interactive dialogue solution includes: Determining the dialogue behavior information of the pickup service object based on the multimodal scenario information, and determining the event task state corresponding to the item pickup event; Based on the pickup interactive dialogue plan, the dialogue behavior information, and the event task status, determining the current task dialogue node, and obtaining the pickup reply content corresponding to the task dialogue node; The pickup reply content corresponding to the current task dialogue node is output to the pickup service object based on the current task dialogue node.

7. The method according to claim 6, characterized in that The outputting the pickup reply content corresponding to the current task dialogue node to the pickup service object based on the current task dialogue node includes: If the current task dialogue node is a pickup process node type, performing the step of determining the dialogue behavior information of the pickup service object based on the multimodal scenario information and determining the event task state corresponding to the item pickup event; If the current task dialogue node is of the pickup completion type, then determine that the pickup service object triggers the item delivery process for the item being delivered on behalf of another, or determine that the pickup service object triggers the item delivery process for the item being delivered on behalf of another and generates task completion information for the item delivery task.

8. The method according to claim 1, characterized in that The method further comprises: Responding to a user's item delivery instruction for an intelligent agent service, obtaining item delivery information; An item delivery task is generated based on the item delivery information.

9. A device for sending items on behalf of others, characterized in that: Applied to smart home devices, the device includes: The monitoring module is used to monitor multimodal scene information in home scenarios based on item delivery tasks; An identification module, configured to identify incoming visit events using an intelligent large model based on the multimodal scene information, obtain item pickup events for pickup service recipients, and generate an interactive dialogue solution for item pickup; The dialogue module is used to output the pickup reply content to the pickup service object based on the pickup interactive dialogue solution, and the pickup reply content is used to instruct the pickup service object to trigger the item delivery process of the item delivery.

10. An electronic device, characterized in that: include: A processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the method steps according to any one of claims 1 to 8.

Images