Article signing processing method and device, storage medium and electronic equipment

Abstract

The invention discloses an article signing processing method and device, a storage medium and electronic equipment, and the method comprises the steps: collecting a target scene image of a scene in front of a door, carrying out the scene recognition based on the target scene image through employing a large model of a guard in front of the door, and obtaining a target article delivery scene; and generating a target article signing process for the target article delivery scene, monitoring at least one article delivery event in the target article delivery scene based on the target article signing process, performing article signing processing based on the article delivery event, and starting an article nursing mode. Therefore, the article delivery event in the scene in front of the door can be processed through the intelligent door lock, the automatic signing process under unattended operation is achieved, the signed article is intelligently nursed, unsafe behaviors such as theft and damage are effectively prevented, and the safety guarantee level of the article is remarkably improved.

Description

Technical Field

[0001] The present application relates to the field of smart home technology, and in particular to a method, device, storage medium and electronic device for processing item receipt. Background Art

[0002] With the rapid development of smart home technology, smart door locks, as an important part of smart home, have gradually entered people's daily life. Smart door locks are favored by the majority of users for their convenience, security and diverse unlocking methods (such as fingerprint, password, face recognition, etc.). At the same time, with the popularization of e-commerce, delivery services have become an indispensable part of daily life, such as delivery services for delivery items such as express delivery and take-out. However, traditional delivery services have security risks. For example, when the user is not at home, the delivered items may be lost or stolen. Summary of the invention

[0003] The embodiments of the present application provide a method, device, storage medium and electronic device for processing item receipt, which can take care of the delivered items and improve the safety of the delivered items.

[0004] In a first aspect, an embodiment of the present application provides a method for processing item receipt, including:

[0005] Collect the target scene image of the scene in front of the door;

[0006] Based on the above target scene image, the door guard large model is used to perform scene recognition to obtain the target item delivery scene, and generate the target item receipt process for the above target item delivery scene;

[0007] Based on the target item receipt process, at least one item delivery event in the target item delivery scenario is monitored, and based on the item delivery event, item receipt processing is performed and an item care mode is started.

[0008] In a possible implementation, the target item receipt process at least includes a delivery event monitoring process, an item receipt process, and an item care process; the target item receipt process monitors at least one item delivery event in the target item delivery scenario, performs item receipt processing based on the item delivery event, and starts an item care mode, including:

[0009] Based on the above-mentioned delivery event monitoring process, monitor the door-front delivery scene image in the target item delivery scene, and determine at least one item delivery event corresponding to the above-mentioned door-front delivery scene image;

[0010] Based on the above-mentioned item receipt process, when the above-mentioned item delivery event is a call event of the item delivery object, an item placement receipt reminder for the item receipt area is output to the item delivery object;

[0011] Based on the above-mentioned item guarding process, when the above-mentioned item delivery event is an item placement completion receipt event, an item guarding mode for the above-mentioned item receipt area is triggered.

[0012] In a possible implementation manner, before outputting the item placement and receipt reminder for the item receipt area to the item delivery object, the method further includes:

[0013] The item care area where the user's preset label is located is determined from the above-mentioned door-to-door delivery scene image, and the above-mentioned item care area is used as the above-mentioned item receipt area.

[0014] In a possible implementation, after outputting the item placement and receipt reminder for the item receipt area to the item delivery object, the method further includes:

[0015] Acquire the target object voice and target object action image for the object of item delivery from the door-to-door delivery scene image;

[0016] Based on the target object's voice and the target object's action image, the door guard model is used to identify the object's intention for item delivery, and a delivery dialogue text is generated based on the object's intention for item delivery;

[0017] The delivery dialogue text is output to the item delivery object.

[0018] In a possible implementation, the target item delivery scene is obtained by using the door guard large model based on the target scene image to perform scene recognition, and a target item receipt process for the target item delivery scene is generated, including:

[0019] Based on the above target scene image, the door guard large model is used to perform scene recognition to obtain the target item delivery scene;

[0020] The door-front guard model is used to determine at least one target item delivery scenario element of the target item delivery scenario, and a target item receipt process for the target item delivery scenario is generated based on the target item delivery scenario element.

[0021] In a possible implementation, the target item receipt process for the target item delivery scenario is generated based on the target item delivery scenario elements, including:

[0022] Determine the target item type, the user's home status, and the delivery time period from the above target item delivery scenario elements;

[0023] Based on the target item type, user's home status and delivery time period, the target item receipt process is queried in the preset receipt scenario process mapping relationship.

[0024] In a possible implementation, the door guard model is trained according to the following steps:

[0025] Obtain a basic large model, and create an initial gate guarding large model for the gate guarding scenario based on the basic large model;

[0026] Acquire a sample scene image of item delivery including a door-front scene, and annotate the sample scene image with an item delivery scene label and an item receipt process label;

[0027] Based on the sample scene images, at least one round of model training is performed on the initial gate guard model;

[0028] In the process of model forward propagation training, the initial door guard model is used to determine the predicted item delivery scene based on the sample scene image, and the item delivery process is generated based on the predicted item delivery scene to obtain the predicted item receipt process;

[0029] During the model back propagation training process, the item delivery scenario prediction loss is determined based on the above-mentioned predicted item delivery scenario and the above-mentioned item delivery scenario label, the item receipt process generation loss is determined based on the above-mentioned predicted item receipt process and the above-mentioned item receipt process label, and the model comprehensive loss is determined based on the above-mentioned item delivery scenario prediction loss and the above-mentioned item receipt process generation loss. The above-mentioned model comprehensive loss is used to adjust the model parameters of the above-mentioned initial door guarding model until the model training of the above-mentioned initial door guarding model is completed, and the above-mentioned door guarding model is obtained.

[0030] In a second aspect, the embodiment of the present application further provides an item receipt processing device, comprising:

[0031] An image acquisition module is used to acquire a target scene image of a scene in front of the door;

[0032] A scene recognition module, which is used to perform scene recognition based on the target scene image using the door front guard large model to obtain the target item delivery scene, and generate a target item receipt process for the target item delivery scene;

[0033] The item receipt processing module is used to monitor at least one item delivery event in the target item delivery scenario based on the target item receipt process, perform item receipt processing based on the above item delivery event, and start the item care mode.

[0034] In a third aspect, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored. When the computer program is run on a computer, the computer executes the method for processing item receipt as provided in any embodiment of the present application.

[0035] In a fourth aspect, an embodiment of the present application further provides an electronic device, comprising a processor and a memory, wherein the memory has a computer program, and the processor calls the computer program to execute the method for processing item receipt as provided in any embodiment of the present application.

[0036] The technical solution provided by the embodiment of the present application collects the target scene image of the scene in front of the door, uses the large model of the door guard to perform scene recognition based on the target scene image to obtain the target item delivery scene, generates the target item receipt process for the target item delivery scene, monitors at least one item delivery event in the target item delivery scene based on the target item receipt process, performs item receipt processing based on the item delivery event, and starts the item care mode. In this way, the present application can process the item delivery event in the scene in front of the door through the smart door lock, realize the automatic receipt process under unattended operation, and perform intelligent care on the signed items, effectively preventing unsafe behaviors such as theft and damage, and significantly improving the security level of items. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0038] Figure 1 A first flow chart of the method for processing item receipt provided in an embodiment of the present application.

[0039] Figure 2 A schematic diagram of a guard area for a door-front scene provided in the item receipt processing method provided in an embodiment of the present application.

[0040] Figure 3 A second flow chart of the method for processing item receipt provided in an embodiment of the present application.

[0041] Figure 4 A schematic diagram of the structure of an item receipt processing device provided in an embodiment of the present application.

[0042] Figure 5 A first structural schematic diagram of an electronic device provided in an embodiment of the present application.

[0043] Figure 6 A second structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0045] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The inclusion of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0046] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0047] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes machine learning (ML) technology, among which deep learning (DL) is a new research direction in machine learning. It is introduced into machine learning to make it closer to the original goal, namely artificial intelligence. At present, deep learning is mainly used in computer vision, natural language processing and other fields.

[0048] Deep learning is the process of learning the inherent laws and representation levels of sample data. The information obtained in the learning process is very helpful for interpreting data such as text, images, and sound. Using deep learning technology and corresponding training data sets, network models that implement different functions can be trained. For example, a gender classification model can be trained based on a training data set, and an image optimization model can be trained based on another training data set.

[0049] With the rapid development of smart home technology, smart door locks, as an important part of smart home, have gradually entered people's daily life. Smart door locks are favored by the majority of users for their convenience, security and diverse unlocking methods (such as fingerprint, password, face recognition, etc.). At the same time, with the popularization of e-commerce, delivery services have become an indispensable part of daily life, such as delivery services for delivery items such as express delivery and take-out. However, traditional delivery services have security risks. For example, when the user is not at home, the delivered items may be lost or stolen.

[0050] In order to solve the above problems, the embodiment of the present application provides a method for processing item receipt, and the execution subject of the method can be the item receipt processing device provided in the embodiment of the present application, or an electronic device integrated with the item receipt processing device, wherein the item receipt processing device can be implemented in hardware or software. Among them, the electronic device is a smart door lock.

[0051] See also Figure 1 , Figure 1 This is a schematic diagram of the first process of the method for processing item receipt provided in the embodiment of the present application. The method for processing item receipt provided in the embodiment of the present application is applied to a smart door lock, and the specific process of the method for processing item receipt can be as follows:

[0052] S110: Collect a target scene image of the scene in front of the door.

[0053] It is understandable that the item receipt processing method provided in the embodiment of the present application is applied to a smart door lock, which can be installed on a door, which may be a home door, a company door, a store entrance, etc. The door scene refers to the surrounding environment of the installation location of the smart door lock (i.e., in front of the door), including but not limited to the road, pedestrians, vehicles, plants, other buildings, etc. in front of the door.

[0054] The smart door lock includes an image sensor, such as a camera, which can be used to capture an image corresponding to a scene in front of the door.

[0055] Among them, the camera can be a monocular camera, a binocular camera and a 3D structured light camera. Specifically, a monocular camera refers to a device that captures image information only through a single lens. A binocular camera collects images through two cameras respectively, and uses an algorithm to perform three-dimensional reconstruction and face recognition. The binocular camera has a high accuracy of liveness detection and can effectively resist attacks by deceptive means such as photos and videos. The 3D structured light camera projects structured light onto the surface of the object to be measured through a projector, and uses a camera to capture structured light images for three-dimensional reconstruction. In addition, the camera 403 can also have a night vision function, for example, imaging processing is performed in a dark or dim environment through infrared light imaging technology.

[0056] The target scene image is also the image of the scene in front of the door captured by the smart door lock.

[0057] S120: Based on the target scene image, a large door-front guard model is used to perform scene recognition to obtain a target item delivery scene, and a target item receipt process for the target item delivery scene is generated.

[0058] Among them, the large door-front guard model is configured to perform scene recognition on the target scene to obtain the target item delivery scene, and generate a target item receipt process for the target item delivery scene.

[0059] In a possible implementation, the door guard model is trained according to the following steps:

[0060] (1) Obtaining a basic large model, and creating an initial door guarding large model for a door guarding scenario based on the basic large model;

[0061] Among them, the basic large models include but are not limited to large models in the field of image recognition processing, such as convolutional neural networks, such as LeNet-5, AlexNet, VGG, Inception series, ResNet (residual network) and DenseNet (densely connected network). Another example is the Transformer model, which is applied in the field of image processing based on the self-attention mechanism. Another example is lightweight convolutional neural networks such as the MobileNet series and EfficientNet.

[0062] (2) obtaining a sample scene image of item delivery including a door-front scene, and labeling the sample scene image with an item delivery scene label and an item receipt process label;

[0063] (3) performing at least one round of model training on the initial gate guard model based on the sample scene images;

[0064] (4) During the model forward propagation training process, the initial door guard model determines the predicted item delivery scene based on the sample scene image, and generates and processes the item delivery process based on the predicted item delivery scene to obtain the predicted item receipt process;

[0065] (5) During the model back propagation training process, the item delivery scenario prediction loss is determined based on the predicted item delivery scenario and the item delivery scenario label, the item receipt process generation loss is determined based on the predicted item receipt process and the item receipt process label, and the model comprehensive loss is determined based on the item delivery scenario prediction loss and the item receipt process generation loss. The model comprehensive loss is used to adjust the model parameters of the initial door guard model until the initial door guard model completes model training and obtains the door guard model.

[0066] Specifically, based on the predicted item delivery scenario and the item delivery scenario label, the first model loss calculation formula is used to calculate the predicted loss of the item delivery scenario; based on the predicted item receipt process and the item receipt process label, the second model loss calculation formula is used to calculate the loss generated by the item receipt process; based on the predicted loss of the item delivery scenario and the loss generated by the item receipt process, the third model loss calculation formula is used to calculate the model comprehensive loss.

[0067] Among them, the model loss calculation formulas such as the first model loss calculation formula, the second model loss calculation formula and the third model loss calculation formula can be fitted by one or more of the contrast loss calculation formula, the cross entropy loss calculation formula, the hinge loss calculation formula and the like in the related technology.

[0068] Optionally, the model training termination conditions of the initial gate guard model may include, for example, the value of the loss function is less than or equal to a preset loss function threshold, the number of iterations reaches a preset number threshold, etc. The specific model training termination conditions can be determined based on actual conditions and are not specifically limited here.

[0069] In one possible implementation, when a target item delivery scene is obtained by using a large door guard model based on a target scene image to perform scene recognition, and a target item receipt process for the target item delivery scene is generated, a large door guard model can be used to perform scene recognition based on the target scene image to obtain the target item delivery scene, and at least one target item delivery scene element of the target item delivery scene can be determined using the large door guard model, and a target item receipt process for the target item delivery scene is generated based on the target item delivery scene elements.

[0070] The item delivery scene here refers to the item delivery scene in front of the door where the smart door lock is located, for example, the scene of the deliveryman delivering items to the door, and more detailed scenes may include: couriers delivering express, takeaways delivering food, medicine deliverymen delivering medicines, food deliverymen delivering vegetables and meat ingredients, daily necessities deliverymen delivering daily necessities, postmen delivering letters, etc. In the embodiment of the present application, the above-mentioned more detailed scenes will be identified through the large model of guarding in front of the door to better determine the elements of the item delivery scene.

[0071] Among them, the item delivery scene elements may include item type, user home status and delivery time period. It can be understood that the user home status refers to whether the user corresponding to the current delivery item is at home.

[0072] Among them, the item receipt process may include the delivery event monitoring process, the item receipt process, and the item care process. Specifically:

[0073] The delivery event monitoring process refers to the process of detecting delivery events, that is, the process of detecting events in which delivery personnel deliver items in a door-front scene. For example, the smart door lock detects the presence of a courier delivering express items based on the target scene image, that is, the delivery event is detected. For another example, the smart door lock detects the presence of a medicine deliveryman delivering medicines based on the target scene image, that is, the delivery event is detected.

[0074] The item signing process refers to the process of signing for the items that need to be signed for in the delivery event after the delivery event occurs. For example, the delivery event is a courier delivering express, then the signing process may be that the courier rings the doorbell, and the smart door lock detects that the doorbell is triggered and outputs a signing prompt message to sign for the delivered express. For another example, the delivery event is that a medicine deliveryman delivers medicines, then the signing process may be that the courier rings the doorbell, and the smart door lock detects that the doorbell is triggered and outputs a signing prompt message to sign for the delivered medicines.

[0075] The item care process refers to the process of taking care of the corresponding items in the item receipt after the item is signed for. For example, if the delivery event is a courier delivering a courier, and the smart doorbell outputs a receipt prompt message, such as the receipt prompt message that you have received the courier, please put it at the door, then the process of taking care of the items is that the smart door lock performs the care (monitoring) of the signed courier placed at the door. For another example, if the delivery event is a drug delivery person delivering the drug, and the smart doorbell outputs a receipt prompt message, such as the receipt prompt message that you have received the drug, please put it at the door, then the process of taking care of the items is that the smart door lock performs the care of the signed drug placed at the door.

[0076] In one possible implementation, when generating a target item receipt process for a target item delivery scenario based on the target item delivery scenario elements, the target item type, the user's home status, and the delivery time period can be determined from the target item delivery scenario elements, and the target item receipt process can be queried in the preset receipt scenario process mapping relationship based on the target item type, the user's home status, and the delivery time period.

[0077] Among them, the preset signing scene process mapping relationship is the correspondence between the item type, the user's home status and the delivery time period and the item signing process. Different item types, user home status and delivery time periods correspond to different item signing processes. It can be understood that the item type refers to the type of delivered item. For example, the item can include daily necessities, food, vegetables, fruits, meat, and medicines. This is just an example, but the item type is not limited to the above examples and is not specifically limited here.

[0078] S130: monitor at least one item delivery event in the target item delivery scenario based on the target item receipt process, perform item receipt processing based on the item delivery event, and start an item care mode.

[0079] In an embodiment of the present application, the target item receipt process at least includes a delivery event monitoring process, an item receipt process, and an item care process. When monitoring at least one item delivery event in the target item delivery scenario based on the target item receipt process, performing item receipt processing and starting the item care mode based on the item delivery event, the door-front delivery scene image in the target item delivery scenario can be monitored based on the delivery event monitoring process to determine at least one item delivery event corresponding to the door-front delivery scene image; based on the item receipt process, when the item delivery event is a call event of the item delivery object, an item placement receipt reminder for the item receipt area is output to the item delivery object; based on the item care process, when the item delivery event is an item placement completion receipt event, the item care mode for the item receipt area is triggered.

[0080] Among them, the item delivery event may include the arrival of the item delivery object, the item delivery object placing the item, the item delivery object ringing the doorbell, the item delivery object knocking on the door, the item delivery object placing the item in the designated item placement area, etc. It should be noted that the item delivery event exemplified in the item receipt processing method provided in this specification is not limited to the item delivery events exemplified above, and the item delivery behavior of the item delivery object in the process of door-to-door delivery can be regarded as the item delivery event, that is, other item delivery events can also be included in the specific implementation, and the embodiments of this specification are not limited to this.

[0081] The object of item delivery is the delivery person who delivers the item. The call event refers to the behavior or words of the object of item delivery ringing the doorbell or knocking on the door so that the user can sign for the express delivery.

[0082] The door front delivery scene image refers to the image acquired when the smart door lock detects an item delivery event in the door front scene. It is understandable that the door front delivery scene image may include multiple item delivery events (delivery events).

[0083] Among them, the item placement completion signature event refers to the delivery person placing the delivered item at the designated location indicated in the signature prompt information output by the smart door lock, indicating that the item placement completion signature event has occurred.

[0084] Among them, the item guarding mode refers to monitoring items placed in the item receipt area.

[0085] In one possible implementation, before outputting an item placement and signing reminder for the item signing area to the item delivery object, the item care area where the user's preset label is located can be determined from the doorstep delivery scene image, and the item care area can be used as the item signing area.

[0086] The preset tag is a tag that is posted by the user within the shooting range of the smart door lock to identify the guarded area. The object guarded area is located by the user preset tag.

[0087] For example, the user preset tag can be a certain area posted on the opposite wall that the smart door lock can capture, and the area of ​​the picture captured by the smart door lock is determined based on the area to determine the item care area. For example, the area corresponding to the user preset tag can be a rectangular area, a circular area, etc. For another example, the user preset tag can be a positioning tag posted on the opposite wall that the smart door lock can capture, and the user of the positioning tag allows the smart door lock to determine an area in the picture around the positioning tag as the item care area based on the positioning tag.

[0088] Please refer to Figure 2 , Figure 2 This is a schematic diagram of a guarded area of ​​a door scene provided in the article receipt processing method provided in the embodiment of the present application. In the embodiment of the present application, a certain area of ​​the picture taken by the smart door lock can also be used as the article guarded area.

[0089] In specific implementation, the present application is not limited by the execution order of the various steps described. If no conflict occurs, some steps can be performed in other orders or simultaneously.

[0090] As can be seen from the above, the method for processing item receipt provided by the embodiment of the present application collects the target scene image of the scene in front of the door, uses the large model of the door guard to perform scene recognition based on the target scene image to obtain the target item delivery scene, generates a target item receipt process for the target item delivery scene, monitors at least one item delivery event in the target item delivery scene based on the target item receipt process, performs item receipt processing based on the item delivery event, and starts the item care mode. In this way, the present application can process the item delivery event in the scene in front of the door through the smart door lock, realize the automatic receipt process under unattended operation, and perform intelligent care on the signed items, effectively preventing unsafe behaviors such as theft and damage, and significantly improving the security level of the items.

[0091] The method described in the above embodiment is further described in detail below with examples.

[0092] See also Figure 3 , Figure 3This is a second flow chart of the method for processing item receipt provided in the embodiment of the present application. The method for processing item receipt provided in the embodiment of the present application is applied to a smart door lock, and the specific flow of the method for processing item receipt can be as follows:

[0093] S210: Collect a target scene image of the scene in front of the door.

[0094] In the embodiment of the present application, a target scene image of the scene in front of the door is collected by a smart door lock.

[0095] Among them, the target scene image is also the image of the scene in front of the door collected by the smart door lock.

[0096] It is understandable that the item receipt processing method provided in the embodiment of the present application is applied to a smart door lock, which can be installed on a door, which may be a home door, a company door, a store entrance, etc. The door scene refers to the surrounding environment of the installation location of the smart door lock (i.e., in front of the door), including but not limited to the road, pedestrians, vehicles, plants, other buildings, etc. in front of the door.

[0097] In a possible implementation, in an embodiment of the present application, when collecting a target scene image of a scene in front of a door, an image corresponding to a preset focal length can be obtained as the target scene image, thereby avoiding obtaining scene images that are irrelevant to the item receipt process in the present application. The preset focal length in the present application can be set by a person skilled in the art according to the needs of the actual scene. For example, generally when the item delivery object is delivering items, the distance in front of the user's door is generally a smaller distance range. Therefore, the scene within this distance range can be photographed, that is, the focal length of the shooting can be adjusted according to this distance range. It can be understood that when shooting, the focal length can be adjusted within this distance range, and different focal lengths can be used to clearly capture the characters, events, etc. in the scene image.

[0098] S220: Based on the target scene image, a large door guard model is used to perform scene recognition to obtain a target item delivery scene.

[0099] Among them, the large door-front guard model is configured to perform scene recognition on the target scene to obtain the target item delivery scene, and generate a target item receipt process for the target item delivery scene.

[0100] The item delivery scene here refers to the item delivery scene in front of the smart door lock, for example, the scene of the deliveryman delivering items at the door. More detailed scenes may include: couriers delivering express, takeaways delivering food, medicine deliverymen delivering medicines, food deliverymen delivering vegetables and meat ingredients, daily necessities deliverymen delivering daily necessities, postmen delivering letters, etc. The above-mentioned item delivery scene can be identified through the above-mentioned front door guard model.

[0101] It can be understood that the target item delivery scene refers to the item delivery scene captured by the current smart door lock.

[0102] S230. Use the door-front guard model to determine at least one target item delivery scene element of the target item delivery scene, and generate a target item receipt process for the target item delivery scene based on the target item delivery scene element. The target item receipt process at least includes a delivery event monitoring process, an item receipt process, and an item care process.

[0103] Among them, the item delivery scenario elements may include items type, user home status, delivery time period and other elements. It should be noted that the item delivery scenario elements used in the item receipt processing method provided in this specification are not limited to the item delivery scenario elements exemplified above, and may also include other item delivery scenario elements in the specific implementation. The embodiments of this specification do not limit this.

[0104] For example, if the target item delivery element detected by the current smart door lock is that the delivery item is a courier, then based on the courier, there is an item receipt process corresponding to the courier delivery scenario.

[0105] For another example, if the target item delivery element detected by the current smart door lock is that the delivery item is food, then based on the food, there is an item receipt process corresponding to the food delivery scenario.

[0106] S240: Monitor the doorstep delivery scene image in the target item delivery scene based on the delivery event monitoring process, and determine at least one item delivery event corresponding to the doorstep delivery scene image.

[0107] As can be seen from the foregoing, the target item receipt process at least includes a delivery event monitoring process, an item receipt process and an item care process. In an embodiment of the present application, the door-front delivery scene image of the target item delivery scene is monitored through the delivery event monitoring process in the target item receipt process, and whether an item delivery event occurs is determined based on the door-front delivery scene image. If an item delivery event is detected, the next process of the delivery event monitoring process in the target item receipt process is carried out.

[0108] S250: Based on the item receipt process, when the item delivery event is an item delivery object call event, output an item placement receipt reminder for the item receipt area to the item delivery object.

[0109] Among them, the item delivery object calling event refers to the behavior of calling the user to sign for the express delivery, such as the item delivery object ringing the doorbell, the item delivery object knocking on the door, and so on.

[0110] For example, based on the item receipt process, when the item delivery event is that the item delivery recipient presses the doorbell of the smart door lock, the smart door lock can issue a voice instruction to the item delivery recipient to place the delivery item in the item receipt area, thereby completing the item receipt process.

[0111] S260: Acquire the target object voice and target object action image for the item delivery object from the door-to-door delivery scene image.

[0112] The target object voice refers to the sound made by the item delivery object, and the target object action image refers to the action performed by the item delivery object.

[0113] In the embodiment of the present application, voice and actions for the item delivery object can also be obtained from the door-to-door delivery scene.

[0114] S270: Based on the target object's voice and the target object's action image, a large door-front guard model is used to identify the object's intention for item delivery, and a delivery dialogue text is generated based on the object's intention for item delivery.

[0115] In the embodiment of the present application, the intention of the item delivery object is identified through the above-mentioned target object voice and target object image, and the delivery dialogue text is generated based on the intention of the item delivery object.

[0116] For example, the recognized target object voice may be "Do you want to put the express here?", and the recognized target object action image may be the object of item delivery pointing to the location where he wants to put the express. The door guard model can identify whether the location of the object's finger is the item receipt area. If so, it can generate a delivery dialogue text of "Yes", and if not, it can generate a delivery dialogue text of "The item receipt area is near the preset label". For another example, if the door guard model recognizes that the object of item delivery makes an OK gesture, the door guard model can generate a delivery dialogue text of "You are so smart, thank you for your hard work!"

[0117] S280: Output the delivery dialogue text to the item delivery object.

[0118] In an embodiment of the present application, after a delivery dialogue text is generated based on the item delivery object intention, the delivery dialogue text can be directly displayed on the display screen of the smart door lock, or the delivery dialogue text can be converted into voice for output.

[0119] S290: Based on the item guarding process, when the item delivery event is an item placement completion receipt event, an item guarding mode for the item receipt area is triggered.

[0120] In the embodiment of the present application, when it is detected that the item delivery event is the placement of the item in the item receipt area, the smart door lock takes care of the items in the item receipt area.

[0121] For example, in the item guarding mode, the smart door lock can detect whether items in the item receipt area are taken away by non-users, or detect whether non-users move near the item receipt area, and send these detected situations to the user's mobile phone, so that the user can understand the safety of the item placement, etc.

[0122] As can be seen from the above, the method for processing item receipt proposed in the embodiment of the present application collects the target scene image of the door scene, uses the door guard large model to perform scene recognition based on the target scene image to obtain the target item delivery scene, uses the door guard large model to determine at least one target item delivery scene element of the target item delivery scene, and generates a target item receipt process for the target item delivery scene based on the target item delivery scene element. The target item receipt process at least includes a delivery event monitoring process, an item receipt process, and an item care process. Based on the delivery event monitoring process, the door delivery scene image in the target item delivery scene is monitored to determine the door delivery scene image corresponding to the target item delivery scene. At least one item delivery event, based on the item receipt process, when the item delivery event is a call event of the item delivery object, an item placement and receipt reminder for the item receipt area is output to the item delivery object, the target object voice and target object action image for the item delivery object are obtained from the door front delivery scene image, the door front guard large model is used to identify the item delivery object intention based on the target object voice and target object action image, a delivery dialogue text is generated based on the item delivery object intention, and the delivery dialogue text is output to the item delivery object, and based on the item care process, when the item delivery event is an item placement completion receipt event, the item care mode for the item receipt area is triggered. In this way, the present application can process the item delivery events in the door front scene through the smart door lock, realize the automatic receipt process under unattended operation, and perform intelligent care on the signed items, effectively preventing unsafe behaviors such as theft and damage, and significantly improving the security level of items.

[0123] In one embodiment, a device for processing item receipt is also provided. Figure 4 , Figure 4 The schematic diagram of the structure of the device 300 for processing item receipt provided in the embodiment of the present application. The device 300 for processing item receipt is applied to an electronic device, and the device 300 for processing item receipt includes an image acquisition module 301, a scene recognition module 302, and an item receipt processing module 303, as follows:

[0124] An image acquisition module 301 is used to acquire a target scene image of a scene in front of a door;

[0125] A scene recognition module 302 is used to perform scene recognition based on the target scene image using the door guard large model to obtain the target item delivery scene, and generate a target item receipt process for the target item delivery scene;

[0126] The item receipt processing module 303 is used to monitor at least one item delivery event in the target item delivery scenario based on the target item receipt process, perform item receipt processing based on the item delivery event, and start the item care mode.

[0127] In a possible implementation, the target item receipt process at least includes a delivery event monitoring process, an item receipt process, and an item care process. The item receipt processing module 303 may be used to:

[0128] Based on the above-mentioned delivery event monitoring process, monitor the door-front delivery scene image in the target item delivery scene, and determine at least one item delivery event corresponding to the above-mentioned door-front delivery scene image;

[0129] Based on the above-mentioned item receipt process, when the above-mentioned item delivery event is a call event of the item delivery object, an item placement receipt reminder for the item receipt area is output to the item delivery object;

[0130] Based on the above-mentioned item guarding process, when the above-mentioned item delivery event is an item placement completion receipt event, an item guarding mode for the above-mentioned item receipt area is triggered.

[0131] In a possible implementation, the item receipt processing module 303 may also be used to:

[0132] The item care area where the user's preset label is located is determined from the above-mentioned door-to-door delivery scene image, and the above-mentioned item care area is used as the above-mentioned item receipt area.

[0133] In a possible implementation, the item receipt processing module 303 may also be used to:

[0134] Acquire the target object voice and target object action image for the object of item delivery from the door-to-door delivery scene image;

[0135] Based on the target object's voice and the target object's action image, the door guard model is used to identify the object's intention for item delivery, and a delivery dialogue text is generated based on the object's intention for item delivery;

[0136] The delivery dialogue text is output to the item delivery object.

[0137] In a possible implementation, the scene recognition module 302 may be used to:

[0138] Based on the above target scene image, the door guard large model is used to perform scene recognition to obtain the target item delivery scene;

[0139] The door-front guard model is used to determine at least one target item delivery scenario element of the target item delivery scenario, and a target item receipt process for the target item delivery scenario is generated based on the target item delivery scenario element.

[0140] In a possible implementation, the scene recognition module 302 may be used to:

[0141] Determine the target item type, the user's home status, and the delivery time period from the above target item delivery scenario elements;

[0142] Based on the target item type, user's home status and delivery time period, the target item receipt process is queried in the preset receipt scenario process mapping relationship.

[0143] In a possible implementation, the door guard model is trained according to the following steps:

[0144] Obtain a basic large model, and create an initial gate guarding large model for the gate guarding scenario based on the basic large model;

[0145] Acquire a sample scene image of item delivery including a door-front scene, and annotate the sample scene image with an item delivery scene label and an item receipt process label;

[0146] Based on the sample scene images, at least one round of model training is performed on the initial gate guard model;

[0147] In the process of model forward propagation training, the initial door guard model is used to determine the predicted item delivery scene based on the sample scene image, and the item delivery process is generated based on the predicted item delivery scene to obtain the predicted item receipt process;

[0148] During the model back propagation training process, the item delivery scenario prediction loss is determined based on the above-mentioned predicted item delivery scenario and the above-mentioned item delivery scenario label, the item receipt process generation loss is determined based on the above-mentioned predicted item receipt process and the above-mentioned item receipt process label, and the model comprehensive loss is determined based on the above-mentioned item delivery scenario prediction loss and the above-mentioned item receipt process generation loss. The above-mentioned model comprehensive loss is used to adjust the model parameters of the above-mentioned initial door guarding model until the model training of the above-mentioned initial door guarding model is completed, and the above-mentioned door guarding model is obtained.

[0149] It should be noted that the item receipt processing device provided in the embodiment of the present application belongs to the same concept as the item receipt processing method in the above embodiment. Any method provided in the item receipt processing method embodiment can be implemented through the item receipt processing device. The specific implementation process is detailed in the item receipt processing method embodiment, which will not be repeated here.

[0150] In addition, in order to better implement the item receipt processing method in the embodiment of the present application, based on the item receipt processing method, the present application also provides an electronic device, which is a smart door lock. Figure 5 , Figure 5 The first structural diagram of the electronic device provided in the embodiment of the present application is as follows: The electronic device 400 includes a processor 401, a memory 402, and a camera 403. The processor 401 is electrically connected to the memory 402.

[0151] The processor 401 is the control center of the electronic device 400. It uses various interfaces and lines to connect various parts of the entire electronic device. By running or calling the computer program stored in the memory 402, and calling the data stored in the memory 402, it executes various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole. Among them, the processor 401 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0152] The memory 402 can be used to store computer programs and data. The computer program stored in the memory 402 contains instructions that can be executed in the processor. The computer program can form various functional modules. The processor 401 executes various functional applications and data processing by calling the computer program stored in the memory 402. Among them, the memory 402 can mainly include a program storage area and a data storage area. The program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the electronic device 400 (such as audio data, video data, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart MediaCard, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0153] The camera 403 can be used to capture images corresponding to the scene in front of the door. The camera can be a monocular camera, a binocular camera, and a 3D structured light camera. Specifically, a monocular camera refers to a device that captures image information only through a single lens. A binocular camera collects images through two cameras respectively, and uses algorithms to perform three-dimensional reconstruction and face recognition. The binocular camera has a high accuracy in live detection and can effectively resist attacks by deceptive means such as photos and videos. The 3D structured light camera projects structured light onto the surface of the object being measured through a projector, and uses a camera to capture structured light images for three-dimensional reconstruction. In addition, the camera 403 can also have a night vision function, for example, imaging processing is performed in a dark or dim environment through infrared imaging technology.

[0154] In this embodiment, the processor 401 in the electronic device 400 loads instructions corresponding to the processes of one or more computer programs into the memory 402 according to the following steps, and the processor 401 runs the computer programs stored in the memory 402 to implement various functions:

[0155] Collect the target scene image of the scene in front of the door;

[0156] Based on the above target scene image, the door guard large model is used to perform scene recognition to obtain the target item delivery scene, and generate the target item receipt process for the above target item delivery scene;

[0157] Based on the target item receipt process, at least one item delivery event in the target item delivery scenario is monitored, and based on the item delivery event, item receipt processing is performed and an item care mode is started.

[0158] In one possible implementation, see Figure 6 , Figure 6 A second structural diagram of an electronic device provided in an embodiment of the present application. The electronic device 400 further includes: a radio frequency circuit 404, a display screen 405, a control circuit 406, an input unit 407, an audio circuit 408, a sensor 409, and a power supply 410. The processor 401 is electrically connected to the radio frequency circuit 404, the display screen 405, the control circuit 406, the input unit 407, the audio circuit 408, the sensor 409, and the power supply 410, respectively.

[0159] The radio frequency circuit 404 is used to send and receive radio frequency signals to communicate with network devices or other electronic devices through wireless communication.

[0160] The display screen 405 may be used to display information input by a user or information provided to a user and various graphical user interfaces of the electronic device. These graphical user interfaces may be composed of images, texts, icons, videos and any combination thereof.

[0161] The control circuit 406 is electrically connected to the display screen 405 and is used to control the display screen 405 to display information.

[0162] The input unit 407 can be used to receive input numbers, character information or user feature information (such as fingerprints), and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control. The input unit 407 can include a fingerprint recognition module.

[0163] The audio circuit 408 can provide an audio interface between the user and the electronic device through a speaker and a microphone. The audio circuit 408 includes a microphone. The microphone is electrically connected to the processor 401. The microphone is used to receive voice information input by the user.

[0164] The sensor 409 is used to collect external environment information. The sensor 409 may include one or more sensors such as an environment brightness sensor, an acceleration sensor, and a gyroscope.

[0165] The power supply 410 is used to supply power to various components of the electronic device 400. In a possible implementation, the power supply 410 may be logically connected to the processor 401 through a power management system, so that the power management system can manage charging, discharging, power consumption, and other functions.

[0166] Although not shown in the figure, the electronic device 400 may also include a camera, a Bluetooth module, etc., which will not be described in detail here.

[0167] In this embodiment, the processor 401 in the electronic device 400 loads instructions corresponding to the processes of one or more computer programs into the memory 402 according to the following steps, and the processor 401 runs the computer programs stored in the memory 402 to implement various functions:

[0168] Collect the target scene image of the scene in front of the door;

[0169] Based on the above target scene image, the door guard large model is used to perform scene recognition to obtain the target item delivery scene, and generate the target item receipt process for the above target item delivery scene;

[0170] Based on the target item receipt process, at least one item delivery event in the target item delivery scenario is monitored, and based on the item delivery event, item receipt processing is performed and an item care mode is started.

[0171] In a possible implementation, the target item receipt process includes at least a delivery event monitoring process, an item receipt process, and an item care process. When the processor 401 executes the target item receipt process to monitor at least one item delivery event in the target item delivery scenario, performs item receipt processing based on the item delivery event, and starts the item care mode, it can execute:

[0172] Based on the above-mentioned delivery event monitoring process, monitor the door-front delivery scene image in the target item delivery scene, and determine at least one item delivery event corresponding to the above-mentioned door-front delivery scene image;

[0173] Based on the above-mentioned item receipt process, when the above-mentioned item delivery event is a call event of the item delivery object, an item placement receipt reminder for the item receipt area is output to the item delivery object;

[0174] Based on the above-mentioned item guarding process, when the above-mentioned item delivery event is an item placement completion receipt event, an item guarding mode for the above-mentioned item receipt area is triggered.

[0175] In a possible implementation, before executing the above-mentioned outputting of the item placement and receipt reminder for the item receipt area to the item delivery object, the processor 401 may further execute:

[0176] The item care area where the user's preset label is located is determined from the above-mentioned door-to-door delivery scene image, and the above-mentioned item care area is used as the above-mentioned item receipt area.

[0177] In a possible implementation, after executing the above-mentioned outputting the item placement and receipt reminder for the item receipt area to the item delivery object, the processor 401 may further execute:

[0178] Acquire the target object voice and target object action image for the object of item delivery from the door-to-door delivery scene image;

[0179] Based on the target object's voice and the target object's action image, the door guard model is used to identify the object's intention for item delivery, and a delivery dialogue text is generated based on the object's intention for item delivery;

[0180] The delivery dialogue text is output to the item delivery object.

[0181] In a possible implementation, when the processor 401 performs the scene recognition based on the target scene image using the door guard large model to obtain the target item delivery scene and generates the target item receipt process for the target item delivery scene, it can execute:

[0182] Based on the above target scene image, the door guard large model is used to perform scene recognition to obtain the target item delivery scene;

[0183] The door-front guard model is used to determine at least one target item delivery scenario element of the target item delivery scenario, and a target item receipt process for the target item delivery scenario is generated based on the target item delivery scenario element.

[0184] In a possible implementation, when the processor 401 generates a target item receipt process for the target item delivery scenario based on the target item delivery scenario elements, the processor 401 may execute:

[0185] Determine the target item type, the user's home status, and the delivery time period from the above target item delivery scenario elements;

[0186] Based on the target item type, user's home status and delivery time period, the target item receipt process is queried in the preset receipt scenario process mapping relationship.

[0187] In a possible implementation, the processor 401 performs the training steps for the above-mentioned door guard model as follows:

[0188] Obtain a basic large model, and create an initial gate guarding large model for the gate guarding scenario based on the basic large model;

[0189] Acquire a sample scene image of item delivery including a door-front scene, and annotate the sample scene image with an item delivery scene label and an item receipt process label;

[0190] Based on the sample scene images, at least one round of model training is performed on the initial gate guard model;

[0191] In the process of model forward propagation training, the initial door guard model is used to determine the predicted item delivery scene based on the sample scene image, and the item delivery process is generated based on the predicted item delivery scene to obtain the predicted item receipt process;

[0192] During the model back propagation training process, the item delivery scenario prediction loss is determined based on the above-mentioned predicted item delivery scenario and the above-mentioned item delivery scenario label, the item receipt process generation loss is determined based on the above-mentioned predicted item receipt process and the above-mentioned item receipt process label, and the model comprehensive loss is determined based on the above-mentioned item delivery scenario prediction loss and the above-mentioned item receipt process generation loss. The above-mentioned model comprehensive loss is used to adjust the model parameters of the above-mentioned initial door guarding model until the model training of the above-mentioned initial door guarding model is completed, and the above-mentioned door guarding model is obtained.

[0193] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. When the computer program runs on a computer, the computer executes the method for processing item receipt as described in any of the above embodiments.

[0194] It should be noted that, a person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the storage medium can include but is not limited to: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0195] In addition, the terms "first", "second", and "third" in this application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or device that includes a series of steps or modules is not limited to the listed steps or modules, but some embodiments also include steps or modules that are not listed, or some embodiments also include other steps or modules inherent to these processes, methods, products, or devices.

[0196] The above is a detailed introduction to the item receipt processing method, device, storage medium and electronic device provided in the embodiments of the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core ideas of the present application; at the same time, for those skilled in the art, according to the ideas of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for processing item receipt, characterized in that: Applied to smart door locks, including: Collect the target scene image of the scene in front of the door; Based on the target scene image, the door guard large model is used to perform scene recognition to obtain the target item delivery scene, and generate the target item receipt process for the target item delivery scene; At least one item delivery event in the target item delivery scenario is monitored based on the target item receipt process, and item receipt processing is performed and an item care mode is started based on the item delivery event.

2. The method according to claim 1, characterized in that: The target item signing process at least includes a delivery event monitoring process, an item signing process, and an item care process; The step of monitoring at least one item delivery event in the target item delivery scenario based on the target item receipt process, performing item receipt processing based on the item delivery event, and starting the item care mode includes: Based on the delivery event monitoring process, a door-front delivery scene image in a target item delivery scene is monitored, and at least one item delivery event corresponding to the door-front delivery scene image is determined; Based on the item receipt process, when the item delivery event is an item delivery object call event, outputting an item placement and receipt reminder for the item receipt area to the item delivery object; Based on the item guarding process, when the item delivery event is an item placement completion and receipt event, an item guarding mode for the item receipt area is triggered.

3. The method according to claim 2, before outputting the item placement and receipt reminder for the item receipt area to the item delivery object, further comprising: The item care area where the user's preset label is located is determined from the door-to-door delivery scene image, and the item care area is used as the item receipt area.

4. The method according to claim 2, after outputting the item placement and receipt reminder for the item receipt area to the item delivery object, further comprising: Acquire the target object voice and target object action image for the object of item delivery from the door-to-door delivery scene image; Based on the target object's voice and the target object's action image, a large door-front guard model is used to identify the object's intention for item delivery, and a delivery dialogue text is generated based on the object's intention for item delivery; The delivery dialogue text is output to the item delivery object.

5. The method according to claim 1, wherein the scene recognition is performed based on the target scene image using the door guard large model to obtain the target item delivery scene, and the target item receipt process for the target item delivery scene is generated, comprising: Based on the target scene image, a large door-front guard model is used to perform scene recognition to obtain a target item delivery scene; A large door-front guard model is used to determine at least one target item delivery scenario element of the target item delivery scenario, and a target item receipt process for the target item delivery scenario is generated based on the target item delivery scenario element.

6. The method according to claim 5, wherein generating a target item receipt process for the target item delivery scenario based on the target item delivery scenario elements comprises: Determining the target item type, the user's home status, and the delivery time period from the target item delivery scenario elements; Based on the target item type, user's home status and delivery time period, the target item receipt process is queried in the preset receipt scenario process mapping relationship.

7. The method according to claim 1, characterized in that The door guard model is trained according to the following steps: Obtaining a basic large model, and creating an initial gate guarding large model for the gate guarding scene based on the basic large model; Acquire a sample scene image of item delivery including a door-front scene, and label the sample scene image with an item delivery scene label and an item receipt process label; Based on the sample scene image, at least one round of model training is performed on the initial door guard model; In the process of model forward propagation training, the initial door guard model determines the predicted item delivery scene based on the sample scene image, and generates the item delivery process based on the predicted item delivery scene to obtain the predicted item receipt process; During the model back-propagation training process, the predicted loss of the item delivery scenario is determined based on the predicted item delivery scenario and the item delivery scenario label, the item receipt process generation loss is determined based on the predicted item receipt process and the item receipt process label, and the model comprehensive loss is determined based on the predicted loss of the item delivery scenario and the item receipt process generation loss. The model comprehensive loss is used to adjust the model parameters of the initial door guard model until the model training of the initial door guard model is completed to obtain the door guard model.

8. An article receipt processing device, characterized in that: include: An image acquisition module is used to acquire a target scene image of a scene in front of the door; A scene recognition module, used to perform scene recognition based on the target scene image using a large door-front guard model to obtain a target item delivery scene, and generate a target item receipt process for the target item delivery scene; The item receipt processing module is used to monitor at least one item delivery event in the target item delivery scenario based on the target item receipt process, perform item receipt processing based on the item delivery event, and start the item care mode.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed on a computer, the computer is enabled to execute the method for processing item receipt as claimed in any one of claims 1 to 7.

10. An electronic device comprising a processor and a memory, wherein the memory stores a computer program, wherein: The processor is used to execute the item receipt processing method as described in any one of claims 1 to 7 by calling the computer program.

Images