Indoor article digital management method and system, electronic equipment and storage medium

Abstract

The embodiment of the invention provides an indoor article digital management method and system, electronic equipment and a storage medium, and belongs to the technical field of artificial intelligence. According to the method, spatial positions and article description data of various articles are determined through indoor environment scanning data, the spatial positions and the article description data of the articles are stored in an associated mode, and then a digital twin model is initialized according to the indoor environment scanning data, the article description data and the spatial positions. Indoor environment scanning data are collected again according to a preset time interval, an environment change area is determined according to the indoor environment scanning data collected again, and whether the change degree of the environment change area exceeds a first threshold value or not is judged; according to the method and the device, the position data of the article in the system is updated in time under the condition that the article position change is obvious, and the article query accuracy of a user is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, system, electronic device and storage medium for digital management of indoor items. Background Art

[0002] In indoor environments such as homes and offices, the storage of items often relies on the memory of family members, making it difficult and inefficient to find items. This is especially true when there are a large number of household items. In addition, the elderly face more serious problems finding items due to memory loss or mobility problems.

[0003] Related technologies have proposed an indoor object positioning system that can manage the location of objects in an indoor environment, making it easier for users to find objects. However, users' daily activities may inadvertently change the storage location of objects. The current method is limited in its adaptability to dynamic environmental changes and cannot update the spatial location of objects stored in the system in a timely manner, resulting in users being unable to accurately find the location of objects. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a method, system, electronic device and storage medium for digital management of indoor items, aiming to timely update the spatial location of items stored in the system and improve the accuracy of user item queries.

[0005] To achieve the above purpose, one aspect of an embodiment of the present application proposes a method for digital management of indoor items, comprising the following steps:

[0006] Collecting indoor environment scanning data, wherein the indoor environment scanning data includes a plurality of environment images;

[0007] Determine the spatial position and item description data of various items according to the indoor environment scanning data, and store the spatial position and item description data of the items in association;

[0008] Initializing a digital twin model according to the indoor environment scanning data, the object description data, and the spatial position;

[0009] Re-collect indoor environment scanning data at preset time intervals;

[0010] Determine the area of ​​environmental change based on the re-collected indoor environment scanning data;

[0011] Determine whether the degree of change in the environmental change area exceeds a first threshold. When the degree of change exceeds the first threshold, update the spatial position of the corresponding object in the environmental change area according to the re-collected indoor environment scanning data, and update the object position in the digital twin model according to the spatial position.

[0012] In some embodiments, the step of determining the spatial position and item description data of various items according to the indoor environment scanning data comprises the following steps:

[0013] Performing object target detection according to the indoor environment scanning data to obtain object detection results, wherein the object detection results include object features in multiple environment images and pixel positions of the objects in the images;

[0014] Performing feature matching based on the object features of each environment image in the object detection result to identify the same object in multiple environment images;

[0015] Integrate the object features of the same object in different environment images to obtain object description data;

[0016] According to the calibrated conversion matrix, the pixel position of the object in the environment image is converted into a spatial position to obtain the spatial position of the object.

[0017] In some embodiments, the associating and storing the spatial location of the item and the item description data comprises the following steps:

[0018] generating an identification code for each of the item description data;

[0019] The item description data, the identification code and the spatial position of each item are stored in association in a local database.

[0020] In some embodiments, the initializing the digital twin model according to the indoor environment scanning data, the object description data and the spatial position comprises the following steps:

[0021] Determine the space structure and space specifications according to the indoor environment scanning data;

[0022] Initialize the three-dimensional space model according to the space structure and space specifications;

[0023] The object is drawn at a corresponding position in the three-dimensional space model according to the object description data and the spatial position of the object to obtain a digital twin model.

[0024] In some embodiments, updating the spatial position of the corresponding object in the environment change area according to the re-collected indoor environment scan data includes the following steps:

[0025] Selecting a plurality of environment images including the environment change area from the re-collected indoor environment scanning data;

[0026] Performing object target detection on the selected multiple environment images, and determining the new spatial position of each object in the environment change area according to the object detection results;

[0027] matching the spatial position of the corresponding object from a local database based on the object description data of the object in the environmental change area;

[0028] The new spatial position of the object is compared with the spatial position stored in the local database to see if they are the same. When the new spatial position of the object is different from the spatial position stored in the local database, the spatial position of the object in the local database is replaced by the new spatial position.

[0029] In some embodiments, the indoor item digital management method further includes the following steps:

[0030] Displaying the digital twin model to obtain private item configuration information, wherein the private item configuration information includes an identification code configured as a private item;

[0031] Based on the identification code in the privacy configuration information, an authority access control policy is set for the item description data and spatial location stored in the local database in association with the identification code.

[0032] In some embodiments, the indoor item digital management method further includes the following steps:

[0033] Obtaining an item query instruction, wherein the item query instruction includes a description of the item to be queried;

[0034] Perform data query according to the item query instruction to determine the spatial location of the target item;

[0035] Generate location description information in a visual or voice form according to the spatial location.

[0036] To achieve the above-mentioned purpose, another aspect of the embodiment of the present application proposes a digital management system for indoor items, including:

[0037] The first module is used to collect indoor environment scanning data, wherein the indoor environment scanning data includes multiple environment images;

[0038] A second module is used to determine the spatial position and item description data of various items according to the indoor environment scanning data, and store the spatial position and item description data of the items in association;

[0039] A third module is used to initialize the digital twin model according to the indoor environment scanning data, the object description data and the spatial position;

[0040] The fourth module is used to re-collect indoor environment scanning data according to a preset time interval;

[0041] The fifth module is used to determine the environment change area based on the re-collected indoor environment scanning data;

[0042] The sixth module is used to determine whether the degree of change in the environmental change area exceeds a first threshold. When the degree of change exceeds the first threshold, the spatial position of the corresponding object in the environmental change area is updated according to the re-collected indoor environment scanning data, and the object position in the digital twin model is updated according to the spatial position.

[0043] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application proposes an electronic device, which includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, and the program implements the method described in the above embodiment when executed by the processor.

[0044] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application proposes a storage medium, which is a computer-readable storage medium used for computer-readable storage, and the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the method described in the above embodiment.

[0045] The digital management method, system, electronic device and storage medium for indoor items proposed in the present application determine the spatial position and item description data of various items through indoor environment scanning data, and store the spatial position and item description data of the items in association, and then initialize the digital twin model based on the indoor environment scanning data, item description data and spatial position. Re-collect the indoor environment scanning data at a preset time interval, determine the environmental change area based on the re-collected indoor environment scanning data, and determine whether the degree of change in the environmental change area exceeds a first threshold. If so, update the spatial position of the corresponding items in the environmental change area and the item position in the digital twin model based on the re-collected scanning data. The present application can detect whether the position of an item has changed in a timely manner, and update the location data of the items in the system in a timely manner when the position of the item has changed significantly, thereby improving the accuracy of user item queries. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is a flow chart of a method for digital management of indoor items provided in an embodiment of the present application;

[0047] Figure 2 yes Figure 1A flowchart of the step of determining the spatial positions and object description data of various objects according to the indoor environment scanning data in step S102;

[0048] Figure 3 yes Figure 1 A flowchart of the step of associating and storing the spatial position of the object and the object description data in step S102;

[0049] Figure 4 yes Figure 1 Flow chart of step S103 in FIG.

[0050] Figure 5 yes Figure 1 Flow chart of step S106 in FIG.

[0051] Figure 6 is a flow chart of a method for digital management of indoor items provided by another embodiment of the present application;

[0052] Figure 7 is a flow chart of a method for digital management of indoor items provided by another embodiment of the present application;

[0053] Figure 8 This is a schematic diagram of a digital management system for indoor items provided in an embodiment of the present application;

[0054] Fig. 9 is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0055] Fig.10 is a schematic diagram of a household robot virtual system provided by an embodiment of the present application;

[0056] Fig.11 is a schematic diagram of the workflow of the household robot provided in an embodiment of the present application;

[0057] Fig.12 It is a schematic diagram of the interaction logic of the household robot functional modules provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0059] It should be noted that, although the functional modules are divided in the system and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0061] First, some nouns involved in this application are analyzed:

[0062] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0063] Image caption generates a natural language description for an image and uses the generated description to help applications understand the semantics expressed in the image visual scene. For example, image captioning can convert image retrieval into text retrieval, which is used to classify images and improve image retrieval results. People usually only need to take a quick glance to describe the details of the image visual scene, while automatically adding descriptions to images is a comprehensive and arduous computer vision task that requires converting the complex information contained in the image into a natural language description. Compared with ordinary computer vision tasks, image captioning not only requires identifying objects from images, but also requires associating the identified objects with natural semantics and describing them in natural language. Therefore, image description requires people to extract deep features of images, associate them with semantic features, and convert them for generating descriptions.

[0064] Based on this, the embodiments of the present application provide a method, system, electronic device and storage medium for digital management of indoor items, which aim to timely update the spatial locations of items stored in the system and improve the accuracy of user item queries.

[0065] The indoor item digital management method, system, electronic device and storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the indoor item digital management method in the embodiments of the present application is described.

[0066] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0067] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0068] The digital management method for indoor items provided in the embodiment of the present application relates to the field of artificial intelligence technology. The digital management method for indoor items provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server side, and can also be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or it can be configured as a server cluster or a distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the digital management method for indoor items, etc., but is not limited to the above forms.

[0069] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0070] Figure 1 is an optional flow chart of the indoor item digital management method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S106.

[0071] Step S101, collecting indoor environment scanning data, wherein the indoor environment scanning data includes a plurality of environment images;

[0072] Step S102, determining the spatial positions and object description data of various objects according to the indoor environment scanning data, and storing the spatial positions and object description data of the objects in association;

[0073] Step S103, initializing the digital twin model according to the indoor environment scanning data, object description data and spatial position;

[0074] Step S104, re-collecting indoor environment scanning data at a preset time interval;

[0075] Step S105, determining the environment change area according to the re-collected indoor environment scanning data;

[0076] Step S106, determine whether the degree of change in the environmental change area exceeds the first threshold. When the degree of change exceeds the first threshold, update the spatial position of the corresponding object in the environmental change area according to the re-collected indoor environment scanning data, and update the object position in the digital twin model according to the spatial position.

[0077] In step S101 of some embodiments, a mobile robot equipped with a distance sensor and a camera can be used to scan the spatial environment to obtain multiple environmental images, thereby forming indoor environment scanning data. Among them, the environmental image can be a depth image, and each pixel in the image contains color information and distance information from the camera, so as to facilitate the subsequent calculation of the spatial position of the object based on the distance information and the spatial position of the camera. In another example, a mobile robot equipped with a binocular camera can be used to scan the spatial environment to obtain multiple environmental images, and the spatial position of the object can be calculated based on the binocular ranging principle.

[0078] Furthermore, each time the spatial environment is scanned, the time of this scan can be recorded, and the time of the next scan can be calculated based on the time of this scan, so as to control the mobile robot to perform the next scan.

[0079] In step S102 of some embodiments, artificial intelligence technology (such as a neural network model) may be used to analyze the indoor environment scan data to identify objects in the image. The image description technology is used to perform feature analysis on the objects in the image to obtain object description data, and then the coordinate conversion technology is used to convert the pixel coordinates of the object in the image into coordinates in a spatial coordinate system to obtain the spatial position of the object. The object description data is data that describes various features of the object, including the color, shape, name, and texture of the object.

[0080] See also Figure 2 In some embodiments, the step of determining the spatial positions and item description data of various items according to the indoor environment scanning data in step S102 may include, but is not limited to, steps S201 to S204:

[0081] Step S201, performing object target detection based on the indoor environment scanning data to obtain object detection results, wherein the object detection results include object features in multiple environment images and pixel positions of the objects in the images;

[0082] Step S202, performing feature matching based on the object features of each environment image in the object detection result to identify the same object in multiple environment images;

[0083] Step S203, integrating the object features of the same object in different environment images to obtain object description data;

[0084] Step S204: convert the pixel position of the object in the environment image into a spatial position according to the calibrated conversion matrix to obtain the spatial position of the object.

[0085] In step S201 of some embodiments, the environmental image in the collected indoor environment scanning data may be subjected to image preprocessing, and the image preprocessing includes but is not limited to denoising, graying, and contrast adjustment. The preprocessed environmental image is input into a trained target detection model for object target detection to obtain an object detection result. Among them, the main task of the target detection model is to accurately detect the target object in a given image or video and determine its position and size. The target detection model can be trained using a deep learning model. After constructing the deep learning model framework, the deep learning model is trained based on an image data set annotated with object boundaries, object features, and object classifications, thereby obtaining a target detection model that can extract features and classify objects from the input image.

[0086] In step S202 of some embodiments, since the same object may appear in the captured images, or two adjacent images respectively include a part of the same object, the embodiments of the present application need to identify the same object in multiple images to obtain a complete feature description of the object. Specifically, a feature matching algorithm can be used to identify the same object in different images. The feature point matching algorithm can achieve image matching by extracting key points and descriptors in the image. For example, feature points can include corner points, edge shapes, regions, and color textures, etc., which can be extracted and described by various algorithms, and then various matching algorithms (such as nearest neighbor matching or RANSAC) are used to find the correspondence between the features of different objects in the two images. For example, when the similarity of the features of two objects in the two images is greater than a certain value, the two objects are considered to be the same object. In some other embodiments, a template matching algorithm can also be used to identify the same object in different images. The template matching algorithm can find objects in the image by comparing a small object template image with the target object image. The algorithm is based on two assumptions: first, the target object image should be similar to the object template image; second, the target object should have a similar spatial distribution in the target image as the template image. The algorithm can match the object template image and the target object image by comparing the pixel values. In different environment images, the target object that can be matched with the same object template image is considered to be the same object.

[0087] In step S203 of some embodiments, after identifying the same object in different environmental images, the object features of the same object in different environmental images can be integrated to obtain object description data. Exemplarily, the features of an object in multiple environmental images can be directly spliced ​​and combined into a data set, and the corresponding image number is marked for each object feature to obtain the object description data. In another example, the complete object image can also be reconstructed based on the object features of the same object in different environmental images, and the object features can be re-analyzed based on the reconstructed object image to obtain the object description data.

[0088] In step S204 of some embodiments, a spatial coordinate system may be customized in the indoor space, and then a transformation matrix from the pixel coordinate system to the spatial coordinate system may be obtained through a coordinate system calibration technique, thereby converting the pixel position of an object in the environmental image into a spatial position in the spatial coordinate system.

[0089] See also Figure 3 In some embodiments, the step of associating and storing the spatial location of the item and the item description data in step S102 may include, but is not limited to, steps S301 to S304:

[0090] Step S301, generating an identification code for each item description data;

[0091] Step S302: Store the item description data, identification code and spatial location of each item in a local database in association with each other.

[0092] In this embodiment, after identifying and classifying the items in the indoor space and determining the item description data and spatial location of the items, a unique identification code can be assigned to each item in the system, and the item description data, identification code and spatial location of each item are associated and stored to mark each item in the indoor space, so that the corresponding items can be accurately found based on the identification code later. In addition, storing the item description data, identification code and spatial location of the item in the local database can improve the privacy of the user's indoor environment data compared to storing the data in the cloud.

[0093] In step S103 of some embodiments, the digital twin model (Digital Twin) is a cutting-edge digital simulation technology that uses digital technology to comprehensively model, simulate and monitor physical objects. The digital twin model refers to the precise simulation of equipment, systems or processes in the physical world through digital means to create a virtual model that completely corresponds to it. In this embodiment, after obtaining indoor environment scanning data and analyzing the scanning data to obtain the item description data and spatial position of each item, a digital twin model identical to the real space environment can be created in the computer virtual space. In the digital twin model, the indoor space structure and decoration (such as windows) in the real space environment are directly mapped based on the indoor environment scanning data. Based on the item description data and spatial position of the item, a three-dimensional virtual item object can be created and placed in the corresponding position of the virtual space, which is convenient for the subsequent dynamic change of the item object in the virtual space.

[0094] See also Figure 4 In some embodiments, step S103 may include but is not limited to steps S401 to S403:

[0095] Step S401, determining the space structure and space specifications according to the indoor environment scanning data;

[0096] Step S402, initializing a three-dimensional space model according to the space structure and space specifications;

[0097] Step S403, drawing the object at a corresponding position in the three-dimensional space model according to the object description data and the spatial position of the object to obtain a digital twin model.

[0098] In this embodiment, the mobile robot can scan the indoor space in the middle area of ​​the room for a week to obtain indoor environment scanning data. According to the shooting angle and distance from the wall of the multiple environmental images collected, the collected data can be analyzed to determine the indoor spatial structure (such as the shape of the room) and spatial specifications (such as the size of the indoor space). Then, according to the spatial structure and spatial specifications, the three-dimensional space model is initialized in the virtual space, and the above-mentioned spatial coordinate system is further configured in the three-dimensional space model. Then, the three-dimensional model object of the object is created according to the object description data, and the three-dimensional model object is drawn to the corresponding position of the three-dimensional space model according to the spatial position of the object, so as to obtain a digital twin model identical to the real indoor environment.

[0099] In step S104 of some embodiments, after analyzing and obtaining the spatial position of the object and the digital twin model of the indoor environment, considering that the position of the indoor object may change, it is necessary to re-collect the indoor environment scanning data at a preset time interval to update the spatial position of the object and the digital twin model stored in the system. Specifically, the system will record the time of each scan, and trigger the next scanning process and object recognition process according to the preset scanning time interval, so as to dynamically monitor whether the distribution of objects in the indoor space has changed, and then update the system data.

[0100] In step S105 of some embodiments, after the mobile robot collects a set of environmental images according to the set scanning program, the environmental images collected twice before and after can be compared to determine the environmental change area, wherein the set scanning program is used to control the shooting position and shooting angle of each shooting of the mobile robot in a scanning process. Exemplarily, the last set of environmental images obtained according to the set scanning program are a1, a2, a3, a4 and a5, and the set of environmental images obtained according to the set scanning program this time are b1, b2, b3, b4 and b5. Since the shooting screens of the environmental images of the same shooting order twice before and after are the same, the image similarity of the environmental images of the same shooting order twice before and after (for example, a2 and b2) can be calculated. If the image similarity is less than the set threshold, it is determined that the area corresponding to the environmental image is the environmental change area.

[0101] In step S106 of some embodiments, the degree of change in the environmental change area can be measured by the corresponding image similarity. The smaller the image similarity, the greater the degree of change. It is determined whether the degree of change in the environmental change area exceeds the first threshold. When the degree of change exceeds the first threshold, it indicates that there is a large change in the items in the environmental change area (for example, the position of the items is moved, the items are added or the items are reduced, etc.). At this time, the spatial position of the corresponding items in the environmental change area is updated according to the re-collected indoor environment scanning data, and the position of the items in the digital twin model is updated according to the spatial position. In this embodiment, when the degree of change in the environmental change area exceeds the first threshold, the spatial position of the item is updated, and the system data can be updated in time according to the real environment, thereby improving the accuracy of subsequent item queries. When the degree of change in the environmental change area does not exceed the first threshold, the surface items are only slightly moved, and the spatial position of the items can be temporarily not recalculated to reduce the system operation pressure.

[0102] In some other embodiments, the environment scanning and object location update process may be triggered after receiving a user update request.

[0103] See also Figure 5 In some embodiments, the step of updating the spatial position of the corresponding object in the environment change area according to the re-collected indoor environment scan data in step S106 may include but is not limited to steps S501 to S504:

[0104] Step S501, selecting a plurality of environment images containing the environment change area from the re-collected indoor environment scanning data;

[0105] Step S502, performing object target detection on the selected multiple environment images, and determining the new spatial position of each object in the environment change area according to the object detection results;

[0106] Step S503, matching the spatial position of the corresponding object from the local database based on the object description data of the object in the environment change area;

[0107] Step S504, comparing the new spatial position of the item with the spatial position stored in the local database to see if they are the same. If the new spatial position of the item is different from the spatial position stored in the local database, the spatial position of the item in the local database is replaced with the new spatial position.

[0108] In this embodiment, multiple environmental images containing environmental change areas are selected from the re-collected indoor environmental scanning data, for example, environmental images corresponding to the environmental change areas are b4 and b5, and object target detection is performed on the selected environmental images to identify various objects in the environmental change area and extract the object description data of the objects, and the new spatial position of each object is calculated based on the object pixel coordinates. Based on the object description data of the objects in the environmental change area, the spatial position of the corresponding object is matched from the local database, and the new spatial position of the object is compared with the spatial position stored in the local database to see if they are the same. When the new spatial position of the object is different from the spatial position stored in the local database, indicating that the position of the object has changed, the spatial position of the object in the local database is replaced with the new spatial position, thereby improving the consistency and real-time performance of the data; when the new spatial position of the object is the same as the spatial position stored in the local database, indicating that the position of the object has not changed, the spatial position of the object in the local database does not need to be updated.

[0109] In some embodiments, for items that cannot be identified or whose positions have changed significantly, the system marks them as abnormal and notifies the user to verify.

[0110] See also Figure 6 In some embodiments, the indoor item digital management method of the embodiment of the present application further includes but is not limited to steps S601 to S602:

[0111] Step S601, displaying the digital twin model to obtain the private item configuration information, wherein the private item configuration information includes an identification code configured as a private item;

[0112] Step S602: Based on the identification code in the privacy configuration information, an access control policy is set for the item description data and spatial location stored in the local database in association with the identification code.

[0113] In this embodiment, the digital twin model is displayed on the visual front end. When the user needs to set privacy for certain items, he can click to select the corresponding items in the digital twin model. The system responds to the user's click operation, determines the identification code of the selected item, and obtains the private item configuration information. Based on the identification code in the privacy configuration information, the permission access control policy is set for the item description data and spatial location stored in the local database associated with the identification code. At the same time, the location of the private item is not displayed in the system twin model. The permission access control policy means that when accessing the data, the user accessing the data needs to pass the system's permission authentication. This embodiment allows users to customize privacy settings for the storage location of key items, so that users have complete control over their own data to prevent it from being queried by others.

[0114] See also Figure 7In some embodiments, the indoor item digital management method of the embodiment of the present application may also include but is not limited to steps S701 to S703:

[0115] Step S701, obtaining an item query instruction, wherein the item query instruction includes a description of the item to be queried;

[0116] Step S702, performing data query according to the item query instruction to determine the spatial location of the target item;

[0117] Step S703: Generate location description information in a visual or voice form according to the spatial location.

[0118] In this embodiment, the user can enter an item query instruction to query the system for the location of the item. The item query instruction may include item description information such as an identification code, item name, or item features. The system performs query matching in the local database according to the item query instruction to determine the spatial location of the target item, and then generates location description information in a visual or voice form according to the spatial location. Exemplarily, the visual location description information may be image information marked (e.g., highlighted) according to the corresponding position of the spatial location in the digital twin model; the location description information in the voice form may be based on the reference items (e.g., cabinets) in the surrounding area of ​​the spatial location, combined with the reference items to form voice text information (e.g., the target item is placed on the cabinet). In addition, if the mobile robot has a mechanical arm, it can also guide the user to the storage location of the item through gestures according to the spatial location, or directly retrieve it to the user. Users can easily obtain item location information through a graphical interface or voice commands, effectively solving common household problems such as difficulty in finding items and inconvenient environmental management.

[0119] According to some embodiments of the present application, taking the indoor item digital management method of the embodiment of the present application applied to a home robot as an example, the workflow of the home robot is described as follows:

[0120] First, refer to Fig.10 The internal system of the home robot includes the item management system control center and the item management system. The item management system control center includes three parts: the end-side AI big model, the environment location update detection and computing power storage. The item management system control center is mainly used to understand the user's voice interaction content, control the triggering of home environment modeling and update, and provide computing power and storage for system modeling and modeling results; the item management system includes a data acquisition and processing module, an item location marking module, an environment model creation module and an environment model update module, which are respectively used to collect and analyze image data of the home environment, mark the location information of items in the home digital twin model, establish a 3D digital twin model of the home environment and update the 3D digital twin model of the home environment.

[0121] Please refer to Fig.11 , the workflow of the home robot is as follows: after starting the home robot, scan the home environment and objects, preprocess the scanned images, mark and update the object positions, create and update the home digital twin model, and then determine whether the object position update conditions are met. If so, return to the step of scanning the home environment and objects. Among them, the marking and updating of the object positions are respectively completed by the object position marking module and the object position information update module. The object position marking module is used to identify and classify objects in the image, assign object identification codes, calculate the three-dimensional coordinates (i.e., spatial positions) of the objects, and associate the three-dimensional coordinates with the identification codes; the object position information update module is used to identify objects with changed positions, update the object coordinates in the home environment digital twin model, and update the position information of the objects in the database. The object position update detection module determines whether the object position update conditions are met. The object position update detection module sets a periodic scanning plan and rescans the home environment to update the object position when the periodic scanning interval is reached, the environment or objects change, or the user manually requests an update.

[0122] According to some embodiments of the present application, the system of the home robot provides users with functions such as structured display of indoor maps, map construction, item management, item organization, and data analysis and suggestions. The interaction logic of each function is as follows: Fig.12 shown.

[0123] According to some embodiments of the present application, the embodiments of the present application have the following beneficial effects:

[0124] Highly automated environmental synchronization: Through the dual-drive mechanism of periodic scanning and environmental changes realized by the home robot, any minor changes in the home environment can be captured in time and the digital twin model can be automatically updated. This level of automation is much higher than the traditional technology that relies on manual recording or simple sensors, which greatly reduces the daily management burden of users and improves the real-time and accuracy of data.

[0125] Accurate object identification and positioning: Combined with the current AI big model, it can automatically identify and classify items in the home, and determine the exact location of items through the robot's precise positioning system to achieve efficient management of items.

[0126] Intelligent interaction and diversified feedback: By integrating advanced natural language processing technology, more natural and intuitive voice interaction is achieved. The robot can understand complex user instructions and provide instant and accurate feedback on the location of items. In addition, the robot's gesture guidance and AR navigation functions provide users with diversified feedback options, enhancing the intuitiveness and ease of use of the interactive experience.

[0127] End-side protection innovation: User data is stored locally on the robot, reducing the risk of data exposure in the cloud.

[0128] See also Figure 8 The present application also provides a digital management system for indoor items, including:

[0129] The first module is used to collect indoor environment scanning data, wherein the indoor environment scanning data includes multiple environment images;

[0130] The second module is used to determine the spatial position and object description data of various objects according to the indoor environment scanning data, and store the spatial position and object description data of the objects in association;

[0131] The third module is used to initialize the digital twin model based on the indoor environment scanning data, object description data and spatial location;

[0132] The fourth module is used to re-collect indoor environment scanning data according to a preset time interval;

[0133] The fifth module is used to determine the environment change area based on the re-collected indoor environment scanning data;

[0134] The sixth module is used to determine whether the degree of change in the environmental change area exceeds the first threshold. When the degree of change exceeds the first threshold, the spatial position of the corresponding object in the environmental change area is updated according to the re-collected indoor environment scanning data, and the object position in the digital twin model is updated according to the spatial position.

[0135] It can be understood that the contents of the above-mentioned embodiments of the digital management method for indoor items are applicable to the embodiments of this system, the functions specifically implemented by the embodiments of this system are the same as those in the above-mentioned embodiments of the digital management method for indoor items, and the beneficial effects achieved are also the same as those achieved by the above-mentioned embodiments of the digital management method for indoor items.

[0136] The embodiment of the present application also provides an electronic device, the electronic device comprising: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, and the program is executed by the processor to realize the above-mentioned indoor item digital management method. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0137] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0138] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0139] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the indoor item digital management method of the embodiment of this application;

[0140] Input / output interface 903, used to implement information input and output;

[0141] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0142] A bus 905 that transmits information between the various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0143] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0144] An embodiment of the present application also provides a storage medium, which is a computer-readable storage medium used for computer-readable storage. The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the above-mentioned digital management method for indoor items.

[0145] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0146] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0147] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0148] The system embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0149] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0150] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0151] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0152] In the several embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or units, which can be electrical, mechanical or other forms.

[0153] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0154] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0155] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0156] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A digital management method for indoor items, characterized in that: The following steps are involved: Collecting indoor environment scanning data, wherein the indoor environment scanning data includes a plurality of environment images; Determine the spatial position and item description data of various items according to the indoor environment scanning data, and store the spatial position and item description data of the items in association; Initializing a digital twin model according to the indoor environment scanning data, the object description data, and the spatial position; Re-collect indoor environment scanning data at preset time intervals; Determine the area of ​​environmental change based on the re-collected indoor environment scanning data; Determine whether the degree of change in the environmental change area exceeds a first threshold. When the degree of change exceeds the first threshold, update the spatial position of the corresponding object in the environmental change area according to the re-collected indoor environment scanning data, and update the object position in the digital twin model according to the spatial position.

2. The indoor item digital management method according to claim 1, characterized in that: Determining the spatial positions and item description data of various items according to the indoor environment scanning data comprises the following steps: Performing object target detection according to the indoor environment scanning data to obtain object detection results, wherein the object detection results include object features in multiple environment images and pixel positions of the objects in the images; Performing feature matching based on the object features of each environment image in the object detection result to identify the same object in multiple environment images; Integrate the object features of the same object in different environment images to obtain object description data; According to the calibrated conversion matrix, the pixel position of the object in the environment image is converted into a spatial position to obtain the spatial position of the object.

3. The indoor item digital management method according to claim 2, characterized in that: The associative storage of the spatial position of the item and the item description data comprises the following steps: generating an identification code for each of the item description data; The item description data, the identification code and the spatial position of each item are stored in association in a local database.

4. The indoor item digital management method according to claim 1, characterized in that: Initializing the digital twin model according to the indoor environment scanning data, the object description data and the spatial position comprises the following steps: Determine the space structure and space specifications according to the indoor environment scanning data; Initialize the three-dimensional space model according to the space structure and space specifications; The object is drawn at a corresponding position in the three-dimensional space model according to the object description data and the spatial position of the object to obtain a digital twin model.

5. The indoor item digital management method according to claim 1, characterized in that: The updating of the spatial position of the corresponding object in the environment change area according to the re-collected indoor environment scanning data comprises the following steps: Selecting a plurality of environment images including the environment change area from the re-collected indoor environment scanning data; Performing object target detection on the selected multiple environment images, and determining the new spatial position of each object in the environment change area according to the object detection results; matching the spatial position of the corresponding object from a local database based on the object description data of the object in the environmental change area; The new spatial position of the object is compared with the spatial position stored in the local database to see if they are the same. When the new spatial position of the object is different from the spatial position stored in the local database, the spatial position of the object in the local database is replaced by the new spatial position.

6. The indoor item digital management method according to claim 3, characterized in that: The indoor article digital management method further comprises the following steps: Displaying the digital twin model to obtain private item configuration information, wherein the private item configuration information includes an identification code configured as a private item; Based on the identification code in the privacy configuration information, an authority access control policy is set for the item description data and spatial location stored in the local database in association with the identification code.

7. The indoor item digital management method according to claim 1, characterized in that: The indoor article digital management method further comprises the following steps: Obtaining an item query instruction, wherein the item query instruction includes a description of the item to be queried; Perform data query according to the item query instruction to determine the spatial location of the target item; Generate location description information in a visual or voice form according to the spatial location.

8. A digital management system for indoor items, characterized in that: include: The first module is used to collect indoor environment scanning data, wherein the indoor environment scanning data includes multiple environment images; A second module is used to determine the spatial position and item description data of various items according to the indoor environment scanning data, and store the spatial position and item description data of the items in association; A third module is used to initialize the digital twin model according to the indoor environment scanning data, the object description data and the spatial position; The fourth module is used to re-collect indoor environment scanning data according to a preset time interval; The fifth module is used to determine the environment change area based on the re-collected indoor environment scanning data; The sixth module is used to determine whether the degree of change in the environmental change area exceeds a first threshold. When the degree of change exceeds the first threshold, the spatial position of the corresponding object in the environmental change area is updated according to the re-collected indoor environment scanning data, and the object position in the digital twin model is updated according to the spatial position.

9. An electronic device, characterized in that: The electronic device includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory. When the program is executed by the processor, the steps of the method described in any one of claims 1 to 7 are realized.

10. A storage medium, the storage medium being a computer-readable storage medium, used for computer-readable storage, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of any one of claims 1 to 7.

Images