Article searching method and device, equipment and storage medium

Abstract

The invention provides an article searching method, comprising: acquiring initial picture data of a target space, and acquiring an article database corresponding to the target space, the article database comprising article data corresponding to each article; based on the initial picture data and an article database, constructing a three-dimensional article map of the target space, the three-dimensional article map including position information of each article; determining position indication information of the to-be-searched article on the basis of the three-dimensional article map based on the position information of the article in response to an article searching request of a user for the to-be-searched article; and sending the position indication information to a terminal device of the user, so that the terminal device displays the position indication information. By generating the three-dimensional article map, the position indication information of the article to be searched can be determined in the three-dimensional article map, the position indication information is displayed through the terminal equipment of the user, and visual and efficient article searching experience can be provided for the user.

Description

Technical Field

[0001] The present invention relates to the technical field of smart home appliances, and in particular to an object searching method, device, equipment and storage medium. Background Art

[0002] In modern family life, as refrigerators become an important place for storing food and drinks, the rational use of their internal space has become particularly critical. However, as the number of items in the refrigerator gradually increases, and various ingredients, drinks, leftovers, etc. are placed in layers, a significant problem emerges, that is, it becomes extremely difficult to find specific items in the obscured part. Therefore, how to accurately find the specific items you need in the refrigerator is an issue that needs to be solved urgently. Summary of the invention

[0003] The embodiment of the present invention provides an item search method, which aims to solve the problem of how to accurately find a specific item in a refrigerator. By acquiring the initial image data of the target space and building an item database, a three-dimensional item map is generated, and the location of the item to be found can be quickly and accurately located through the three-dimensional item map, and the location indication information of the item to be found can be determined. The location indication information is displayed through the user's terminal device, which can provide the user with an intuitive and efficient item search experience.

[0004] In a first aspect, an embodiment of the present invention provides an object search method, the method comprising:

[0005] Acquire initial image data of the target space, and acquire an object database corresponding to the target space, wherein the object database includes object data corresponding to each object;

[0006] Based on the initial image data and the object database, construct a three-dimensional object map of the target space, wherein the three-dimensional object map includes location information of each of the objects;

[0007] In response to a user's request for finding an object to be found, based on the location information of the object, determining location indication information of the object to be found on the basis of the three-dimensional object map;

[0008] The location indication information is sent to the terminal device of the user, so that the terminal device displays the location indication information.

[0009] Optionally, the step of obtaining an object database corresponding to the target space includes:

[0010] Obtaining the user's online shopping order information;

[0011] Determining first item data that needs to be placed in the target space according to the online shopping order information;

[0012] Acquire second item data uploaded by the user;

[0013] An object database corresponding to the target space is constructed based on the first object data and the second object data.

[0014] Optionally, the step of constructing a three-dimensional object map of the target space based on the initial image data and the object database includes:

[0015] Performing feature extraction processing on the initial image data to obtain object features corresponding to each object;

[0016] Based on the object features, determining unobstructed objects and obstructed objects in the object database;

[0017] Determine, based on the object data corresponding to the unobstructed object and the object data corresponding to the obstructed object, a first three-dimensional target state corresponding to the unobstructed object and a second three-dimensional target state corresponding to the obstructed object;

[0018] A three-dimensional object map of the target space is constructed based on the first three-dimensional target state, the second three-dimensional target state and the initial image data.

[0019] Optionally, the step of determining a first three-dimensional target state corresponding to the unobstructed object and a second three-dimensional target state corresponding to the obstructed object based on the object data corresponding to the unobstructed object and the object data corresponding to the obstructed object includes:

[0020] Performing three-dimensional reconstruction on the object data corresponding to the unobstructed object to obtain a first three-dimensional model corresponding to the unobstructed object;

[0021] and, performing three-dimensional reconstruction on the object data corresponding to the obscured object to obtain a second three-dimensional model corresponding to the obscured object;

[0022] Based on the first three-dimensional model and the second three-dimensional model, a first three-dimensional target state corresponding to the unobstructed object and a second three-dimensional target state corresponding to the obstructed object are obtained respectively.

[0023] Optionally, the step of constructing the three-dimensional object map of the target space based on the first three-dimensional target state, the second three-dimensional target state and the initial image data includes:

[0024] Based on the first three-dimensional target state and the initial image data, construct an initial three-dimensional object map of the target space, wherein the initial three-dimensional object map includes occupied space and unoccupied space, and the occupied space is occupied by the unobstructed objects;

[0025] Based on the second three-dimensional target state, the blocked object is spatially matched in the unoccupied space to obtain a three-dimensional object map of the target space.

[0026] Optionally, the step of spatially matching the blocked object in the unoccupied space based on the second three-dimensional target state to obtain a three-dimensional object map of the target space includes:

[0027] Based on the second three-dimensional target state, determining a required space corresponding to the obstructed object;

[0028] Based on the required space corresponding to the obstructed object, the obstructed object is spatially matched in the unoccupied space to obtain a three-dimensional object map of the target space.

[0029] Optionally, the step of determining, in response to a user's request for finding an object to be found and based on the location information of the object, the location indication information of the object to be found on the three-dimensional object map comprises:

[0030] After receiving a search request from a user for an object to be found, searching the three-dimensional object map for a target object corresponding to the object to be found;

[0031] Based on the location information of the object, determining the target location information corresponding to the target object in the three-dimensional object map;

[0032] Based on the target location information, location indication information indicating the object to be found is generated.

[0033] In a second aspect, an embodiment of the present invention further provides an object finding device, the object finding device comprising:

[0034] An acquisition module, used to acquire initial image data of a target space, and acquire an object database corresponding to the target space, wherein the object database includes object data corresponding to each object;

[0035] A construction module, configured to construct a three-dimensional object map of the target space based on the initial image data and the object database, wherein the three-dimensional object map includes location information of each of the objects;

[0036] A processing module, for responding to a user's request for finding an object to be found, and determining, based on the location information of the object and on the three-dimensional object map, location indication information of the object to be found;

[0037] The sending module is used to send the location indication information to the terminal device of the user, so that the terminal device can display the location indication information.

[0038] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the object search method provided in the embodiment of the present invention when executing the computer program.

[0039] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the object search method provided in the embodiment of the invention are implemented.

[0040] In an embodiment of the present invention, initial image data of a target space is obtained, and an item database corresponding to the target space is obtained, wherein the item database includes item data corresponding to each item; based on the initial image data and the item database, a three-dimensional item map of the target space is constructed, wherein the three-dimensional item map includes location information of each item; in response to a user's request to find an item to be found, based on the item's location information, location indication information of the item to be found is determined on the basis of the three-dimensional item map; the location indication information is sent to the user's terminal device, and the terminal device displays the location indication information. The present invention generates a three-dimensional item map by obtaining initial image data of the target space and constructing an item database, and can quickly and accurately locate the location of the item to be found through the three-dimensional item map, and determine the location indication information of the item to be found, and display the location indication information through the user's terminal device, which can provide the user with an intuitive and efficient item search experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 is a flow chart of an object search method provided by an embodiment of the present invention;

[0043] Figure 2is a schematic structural diagram of an object finding device provided by an embodiment of the present invention;

[0044] Figure 3 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] like Figure 1 As shown, Figure 1 This is a method flow chart of a method for finding an item provided by an embodiment of the present invention. The method for finding an item comprises the following steps:

[0047] 101. Obtain initial image data of the target space, and obtain an object database corresponding to the target space.

[0048] In an embodiment of the present invention, the above-mentioned object search method can be deployed in an electronic device, and the above-mentioned electronic device can be a server, a gateway device or a user's terminal device, etc. The above-mentioned object search method should be used in furniture or household appliances with object storage space, such as lockers, refrigerators, etc. This embodiment takes a refrigerator as an example for explanation, and other household appliances with object storage space can refer to the relevant embodiments of the refrigerator.

[0049] The target space is a space for storing items, which may be the internal space of a refrigerator. The initial image data may be obtained by photographing the user's terminal device, and the item database includes item data corresponding to each item. Preferably, in this embodiment, the refrigerator is a refrigerator without a camera, and the initial image data is obtained by photographing the user's terminal device. When the user opens the refrigerator door, the user photographs the internal space of the refrigerator through the camera of the terminal device (such as a mobile phone) to obtain the initial image data. The initial image data includes the space image data of the refrigerator and the item image data.

[0050] The above-mentioned item database is a dynamic database. When the quantity of items in the target space changes, the item database will be updated. The quantity change includes adding items or reducing items. For example, when the user puts items in the refrigerator, the quantity change is to increase the items. When the user takes items out of the refrigerator, the quantity change is to reduce the items.

[0051] The above-mentioned item database includes item data corresponding to each item. The above-mentioned item data may include data such as item name, item shape, item color, item size, etc. The above-mentioned item data may also include feature values ​​of the item. The above-mentioned feature values ​​can be obtained by feature extraction based on the item data.

[0052] The item data in the above item database may be uploaded by the user himself or extracted based on the user's shopping information.

[0053] 102. Based on the initial image data and the object database, a three-dimensional object map of the target space is constructed.

[0054] In an embodiment of the present invention, the two-dimensional initial image data can be converted into a three-dimensional space map through a three-dimensional reconstruction method, so that the user can more intuitively understand the spatial distribution of each object in the target space. In the process of constructing a three-dimensional object map, the initial image data can be subjected to feature extraction processing, and the feature information of each object in the image can be extracted through an image recognition algorithm. Then, according to the feature information of the object, the corresponding object is matched and identified in the object database, and the identified object in the object database is determined as an unobstructed object, and the unidentified object in the object database is determined as an obstructed object. The above-mentioned unobstructed object can be understood as not being obstructed as a whole or not being completely obstructed, wherein not being completely obstructed can be understood as not being obstructed in a characteristic part, and therefore, it can be matched and identified in the object database. The obstructed object can be understood as being obstructed as a whole or being obstructed in a characteristic part, and therefore, it cannot be matched and identified in the object database.

[0055] For unobstructed objects, their object data and initial image data can be directly used for 3D reconstruction to generate a 3D model corresponding to the unobstructed object and determine its position and posture in the 3D space; for obstructed objects, since their features are partially or completely obstructed by other objects, more complex algorithms are needed for processing, or object data in the object database can be directly used for 3D reconstruction to generate a 3D model corresponding to the obstructed object. The corresponding position and posture of the 3D model corresponding to the above-mentioned unobstructed object can be identified and determined by image data, and the corresponding position and posture of the 3D model corresponding to the above-mentioned unobstructed object can be determined by performing position and posture prediction processing on its own 3D model and the 3D model of the unobstructed object in the target space.

[0056] At the same time, the target space can also be modeled in three dimensions to obtain a basic space model of the target space, which can also be called a base model. After obtaining the three-dimensional models of all objects, the three-dimensional models of the objects are positioned on the base model according to the location information of each object, and combined to form a three-dimensional object map. The three-dimensional object map includes the location information of each object, and the location information can represent the location of the object in the target space.

[0057] 103. In response to a user's request to find an object to be found, based on the location information of the object, determine location indication information of the object to be found on the basis of the three-dimensional object map.

[0058] In the embodiment of the present invention, after the three-dimensional object map of the target space is constructed, a user's search request may be received, and the search request may be initiated by the user through the terminal device. The search request may include the object data that the user wants to find, that is, the object data of the object to be found, and the object data of the object to be found may include the name of the object and other data, such as the date of the object, the color of the object, the size of the object, etc.

[0059] The above-mentioned lost item request may be data of the type of voice, text, or picture.

[0060] After receiving the request, the target object that matches the user's desired object can be searched in the constructed three-dimensional object map. After finding the target object in the three-dimensional object map, the specific location information of the target object in the three-dimensional map can be determined, and corresponding location indication information can be generated based on the location information of the target object. The above-mentioned location indication information can be a specific spatial coordinate, a direction arrow pointing to the target object, a text description describing the location of the target object, a voice description describing the location of the target object, etc.

[0061] 104. Send the location indication information to a terminal device of the user, so that the terminal device displays the location indication information.

[0062] In an embodiment of the present invention, after obtaining the location indication information of the target object, the location indication information can be sent to the user's terminal device, such as a smart phone, smart watch, smart bracelet or tablet computer, so that the user can intuitively understand the location of the target object and quickly find the target object.

[0063] In a possible embodiment, the types of items and the frequency of searching can be obtained based on the user's search records, and suggestions for the placement and management of items can be recommended based on the user's usage habits and the size of vacant spaces.

[0064] In an embodiment of the present invention, initial image data of a target space is obtained, and an item database corresponding to the target space is obtained, wherein the item database includes item data corresponding to each item; a three-dimensional item map of the target space is constructed based on the initial image data and the item database, wherein the three-dimensional item map includes location information of each item; in response to a user's request to find an item to be found, location indication information of the item to be found is determined based on the three-dimensional item map; the location indication information is sent to the user's terminal device, and the terminal device displays the location indication information. The present invention generates a three-dimensional item map by obtaining initial image data of the target space and constructing an item database, and can quickly and accurately locate the location of the item to be found through the three-dimensional item map, and determine the location indication information of the item to be found, and display the location indication information through the user's terminal device, thereby providing the user with an intuitive and efficient item search experience.

[0065] It should be noted that the object search method provided in the embodiment of the present invention can be applied to devices such as refrigerators, smart phones, tablets, computers, servers, etc. that can perform the object search method.

[0066] It is understandable that in the specific implementation of this application, user data, item data, order data and other related data are involved. When the embodiments in this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data, as well as model training, deployment and use, must comply with relevant laws, regulations and standards of relevant countries and regions.

[0067] Optionally, in the step of obtaining the item database corresponding to the target space, the user's online shopping order information can be obtained; based on the online shopping order information, the first item data that needs to be placed in the target space is determined; the second item data uploaded by the user is obtained; and based on the first item data and the second item data, the item database corresponding to the target space is constructed.

[0068] In the embodiment of the present invention, the user's online shopping order information can be obtained. Considering the popularity of modern e-commerce, online shopping order information has become an important record of the user's purchase of items. By obtaining the user's online shopping order information, the data of items purchased by the user within a period of time can be obtained, thereby determining the items that may need to be placed in the target space.

[0069] The above-mentioned shopping order information requires user authorization to obtain the corresponding online shopping order information from the shopping software. Of course, in some possible embodiments, the above-mentioned online shopping order information can be that the user takes a screenshot of the order page, uploads the screenshot, and performs image recognition on the screenshot to obtain the user's online shopping order information.

[0070] The online shopping order information may include a picture description of the item (multiple pictures), product specifications (size), storage method (refrigerated or frozen), etc. Based on the online shopping order information, the first item data is analyzed and sorted into multiple first item data; wherein, the item data corresponding to the item to be put into the refrigerator can be extracted from the online shopping order information according to the item storage method to form the first item data;

[0071] According to the online shopping order information, the first item data that needs to be placed in the target space can be determined. It is understandable that the first item data mainly refers to items purchased by users through online shopping channels and expected to be placed in the target space. The item data may include data such as item name, item shape, item color, item size, etc. The item data may also include feature values ​​of the item, which may be obtained by feature extraction based on the item data. Of course, in some possible embodiments, the item data may also include information such as item quantity and purchase time.

[0072] In addition to obtaining item data through online shopping order information, you can also obtain second item data uploaded by users. The above second item data mainly refers to item data obtained by users through other channels and manually uploaded to the system. For example, for some items that are not purchased online, or items that users think need to be specially recorded, users can directly upload the item data of the corresponding items as second item data.

[0073] In a possible embodiment, an interface for manually inputting items can be provided to the user on the terminal device to obtain data of multiple second items placed in the refrigerator in addition to the items in the shopping software; wherein, in the interface for manually inputting items, item identification can be performed directly by taking photos, and the item name, item color, item size, outer packaging picture, outer packaging color, etc. can be input.

[0074] After obtaining the first item data and the second item data, an item database corresponding to the target space can be constructed. The above-mentioned item database can be understood as a data table that centrally stores and manages item data, which can integrate and standardize item data from various sources, and provide unified data support for subsequent three-dimensional item map construction and item search.

[0075] In a possible embodiment, the Internet of Things technology can be used to obtain the usage and status changes of the objects in the target space in real time; or the target space can be scanned regularly through image recognition technology to automatically identify and update the object information in the database. For example, when the number of objects in the target space changes, the object database will be updated, and the change in quantity includes adding or reducing objects.

[0076] Through online shopping order information and user uploaded data, a comprehensive, accurate and efficient item database can be built, thereby improving the accuracy of the three-dimensional item map.

[0077] Optionally, in the step of constructing a three-dimensional object map of the target space based on the initial image data and the object database, feature extraction processing may be performed on the initial image data to obtain object features corresponding to each object; based on the object features, unobstructed objects and obscured objects may be determined in the object database; based on the object data corresponding to the unobstructed objects and the object data corresponding to the obscured objects, a first three-dimensional target state corresponding to the unobstructed objects and a second three-dimensional target state corresponding to the obscured objects may be determined; based on the first three-dimensional target state, the second three-dimensional target state and the initial image data, a three-dimensional object map of the target space may be constructed.

[0078] In the embodiment of the present invention, feature extraction processing can be performed on the initial image data, and specifically, feature detection algorithms can be used for feature extraction processing, such as SIFT (Scale Invariant Feature Transform), SURF (Speeded Up Robust Features) or ORB (Oriented BRIEF and Rotation Invariant), etc., to extract feature points of each object in the image. The above feature points can uniquely identify different objects in the image and are used to represent the corresponding objects.

[0079] After the initial image data is processed for feature extraction, the feature information of each item in the image is extracted through an image recognition algorithm. Then, according to the feature information of the item, the corresponding item is matched and identified in the item database, and the items identified in the item database are determined as unobstructed items, and the items not identified in the item database are determined as obstructed items. The above-mentioned unobstructed items can be understood as not being obstructed as a whole or not being completely obstructed, wherein not being completely obstructed can be understood as not being obstructed in the feature part, and therefore, they can be matched and identified in the item database. Obstructed items can be understood as being obstructed as a whole or having their feature parts obstructed, and therefore, cannot be matched and identified in the item database.

[0080] After determining the unobstructed objects and the obstructed objects, for the unobstructed objects, since their features are complete, their three-dimensional shape and posture can be restored through the initial image data combined with three-dimensional reconstruction technology (such as 3D reconstruction based on structured light, stereo vision, etc.), thereby obtaining the first three-dimensional target state.

[0081] For occluded objects, the difficulty of 3D reconstruction increases because the shape information of the characteristic part is missing. However, by combining the corresponding object data in the object database, the possible shape and position of the occluded part can be estimated, thereby constructing a second 3D model and determining the second 3D target state.

[0082] After determining the 3D target states of unobstructed and obstructed objects, the initial image data can be combined to construct a 3D object map of the target space. The 3D object map contains not only the 3D shape and position information of the objects, but also the relative relationship and spatial layout between the objects.

[0083] In one possible embodiment, the initial image can be analyzed to extract the unobstructed features corresponding to the exposed parts of the obstructed objects. For example, the objects in the refrigerator are of different sizes and heights. The exposed parts of the objects placed at the back are the key features for identifying the objects. The unobstructed features are compared with the object data in the object database, which can be the color of the object or the outer packaging, shape, etc. The object with the highest similarity is regarded as the object that is not completely obstructed, and its position outline is also calibrated. For completely obstructed objects, the directly identifiable objects can be filtered out through the object data in the database. After filtering out the incompletely obstructed objects, the remaining objects are the completely obstructed objects.

[0084] By combining the initial image data and three-dimensional reconstruction technology, we can obtain the first three-dimensional target state corresponding to the unobstructed objects and the second three-dimensional target state corresponding to the obstructed objects. Using the first three-dimensional target state corresponding to the unobstructed objects and the second three-dimensional target state corresponding to the obstructed objects, we can construct an accurate three-dimensional object map.

[0085] Optionally, in the step of determining a first three-dimensional target state corresponding to the unobstructed object and a second three-dimensional target state corresponding to the obstructed object based on the object data corresponding to the unobstructed object and the object data corresponding to the obstructed object, the object data corresponding to the unobstructed object can be three-dimensionally reconstructed to obtain a first three-dimensional model corresponding to the unobstructed object; and the object data corresponding to the obstructed object can be three-dimensionally reconstructed to obtain a second three-dimensional model corresponding to the obstructed object; based on the first three-dimensional model and the second three-dimensional model, the first three-dimensional target state corresponding to the unobstructed object and the second three-dimensional target state corresponding to the obstructed object can be obtained respectively.

[0086] In an embodiment of the present invention, the above-mentioned three-dimensional reconstruction technology can be used to restore the three-dimensional shape and structure of an object from two-dimensional image data. Specifically, a variety of algorithms and technologies, such as three-dimensional reconstruction based on feature point matching, stereoscopic vision reconstruction or structured light three-dimensional reconstruction, can be used to construct an accurate three-dimensional model of the unobstructed object, that is, a first three-dimensional model. The first three-dimensional model may include the shape information of the object, and may also include details such as texture and color, which helps to more comprehensively describe the state of the object in the three-dimensional space.

[0087] Since the features of the occluded object in the image are partially or completely covered by other objects, the 3D reconstruction technology based on deep learning can be used to infer the shape and structure of the occluded part from the limited object data, and then obtain a second 3D model of the occluded object.

[0088] After obtaining the first three-dimensional model and the second three-dimensional model, the three-dimensional target states corresponding to the unobstructed objects and the obstructed objects are determined respectively. The above three-dimensional target state can be a comprehensive description of information such as the position, orientation and posture of the object in the three-dimensional space. For unobstructed objects, since their three-dimensional models are more accurate, their three-dimensional target states can be directly determined by comparing and analyzing the initial image data. For example, feature point matching technology can be used to align the three-dimensional model with the objects in the picture to obtain their accurate position and orientation information. For obstructed objects, the prior knowledge in the object database can be used to infer the possible position and posture of the obstructed objects. For example, if an object is usually placed in a specific position or in a specific way, even if it is partially obstructed, its current three-dimensional target state can be inferred based on these prior knowledge.

[0089] In a possible embodiment, it is also possible to consider using multi-view image data to assist in determining the three-dimensional target state of the obscured object. By photographing the target space from different angles, more visible information about the obscured object can be obtained. The above visible information can be integrated and utilized to improve the accuracy of three-dimensional reconstruction and state determination.

[0090] In a possible embodiment, constructing a three-dimensional target state may include the following steps:

[0091] S1. Data collection.

[0092] Multi-view images: The image description in the order can provide photos from as many viewpoints as possible, such as the front, side, top, etc.

[0093] Geometric information: The length, width, height and other information in the product specifications in the order can provide the corresponding geometric information.

[0094] S2. Data processing.

[0095] Image calibration: The collected photos are calibrated to ensure geometric consistency between them.

[0096] Feature extraction: Extract feature points from photos that can be used for 3D reconstruction, such as corners, edges, etc.

[0097] Depth estimation: Using the geometric information, feature point matching and stereo vision technology mentioned above, the depth information of each pixel in the photo is estimated. This can be achieved through single view depth estimation, binocular system or structured light system.

[0098] S3. Three-dimensional modeling.

[0099] Point cloud generation: Based on the depth information, the pixels in the two-dimensional image are converted into points in three-dimensional space to form point cloud data.

[0100] Surface reconstruction: Use point cloud data to construct the surface of a three-dimensional model through algorithms (such as Delaunay triangulation, Poisson surface reconstruction, etc.).

[0101] Texture mapping: Map the texture information in the original photo onto the surface of the 3D model to increase the realism of the model.

[0102] S4. Rendering and visualization.

[0103] Rendering engine: Use professional rendering engines (such as Blender, Maya, etc.) to render three-dimensional models and simulate lighting, shadow and other effects.

[0104] Visualization: Display the rendered 3D model in an appropriate manner, such as through virtual reality (VR), augmented reality (AR) or a regular 3D viewer.

[0105] By deeply processing and analyzing the object data of unobstructed and obstructed objects, accurate three-dimensional models and three-dimensional target states are generated. Using the three-dimensional target state for modeling can further improve the accuracy of the three-dimensional object map.

[0106] Optionally, in the step of constructing a three-dimensional object map of the target space based on the first three-dimensional target state, the second three-dimensional target state and the initial image data, an initial three-dimensional object map of the target space can be constructed based on the first three-dimensional target state and the initial image data, the initial three-dimensional object map including occupied space and unoccupied space, the occupied space being occupied by unobstructed objects; based on the second three-dimensional target state, the obstructed objects are spatially matched in the unoccupied space to obtain the three-dimensional object map of the target space.

[0107] In the embodiment of the present invention, the first three-dimensional target state refers to a comprehensive expression of information such as the position, posture and shape of the unobstructed object in the three-dimensional space; the second three-dimensional target state may be a comprehensive expression of information such as the position, posture and shape of the obstructed object in the three-dimensional space. The initial image data provides an overall visual description of the target space and is the basic data for constructing a three-dimensional object map.

[0108] The construction process first creates an initial three-dimensional object map of the target space based on the first three-dimensional target state and the initial image data. The shape and position of the unobstructed objects can be accurately restored in the three-dimensional space, thereby constructing the initial three-dimensional object map. The initial three-dimensional object map includes the three-dimensional space corresponding to the target space and the three-dimensional models corresponding to the unobstructed objects, wherein the three-dimensional space occupied by the three-dimensional models corresponding to the unobstructed objects is the occupied space, and the three-dimensional space not occupied by the three-dimensional models corresponding to the unobstructed objects is the unoccupied space.

[0109] The second 3D target state is used to spatially match the occluded object in the unoccupied space of the initial 3D object map. Since the occluded object is completely invisible in the initial image data, its position and posture in the 3D space can be determined by the second 3D target state to ensure that the occluded object can be accurately placed in the unoccupied space and maintain a reasonable spatial relationship with other objects.

[0110] Specifically, a deep learning-based spatial matching algorithm can be used. The deep learning-based spatial matching algorithm can automatically infer the reasonable position of the obscured object in the three-dimensional space by learning the spatial relationship patterns in a large amount of sample data. At the same time, the matching results can be further optimized by combining traditional geometric constraints, such as the relative distance and angle between objects, to improve the accuracy and reliability of the three-dimensional object map.

[0111] Optionally, in the step of spatially matching the obstructed object in an unoccupied space based on the second three-dimensional target state to obtain a three-dimensional object map of the target space, the required space corresponding to the obstructed object can be determined based on the second three-dimensional target state; based on the required space corresponding to the obstructed object, the obstructed object is spatially matched in the unoccupied space to obtain a three-dimensional object map of the target space.

[0112] In an embodiment of the present invention, the space required for the obstructed object can be determined based on the second three-dimensional target state. The actual size, shape, and possible spatial posture of the obstructed object are taken into account. Through accurate calculation of the second three-dimensional target state, a minimum spatial volume surrounding the obstructed object can be obtained, that is, the space required for the obstructed object. After determining the space required for the obstructed object, spatial matching can be performed in the unoccupied space of the initial three-dimensional object map. Through spatial matching, a suitable position is found to place the obstructed object so that it does not conflict with other objects in the map and conforms to the actual spatial layout.

[0113] After finding the matching position, the blocked object can be placed at the matching position, and the three-dimensional object map is updated to obtain the three-dimensional object map of the target space.

[0114] By clarifying the second 3D target state of the occluded object and determining its required space, a suitable location can be found in the unoccupied space to place the occluded object, thereby improving the information of the 3D object map and making the 3D object map more accurate.

[0115] In a possible embodiment, after a three-dimensional map is generated for identifiable objects and objects that are not completely obscured based on specification information and initial image data, there will be blank positions (unoccupied space) in the three-dimensional map. When there are multiple blank positions, the bottom areas calculated based on the specification size (size or shape size) of the completely obscured objects are matched with the blank positions respectively. If the error is within a range, it is considered that the completely obscured object a is placed at the blank position A. Finally, an optimized three-dimensional map is generated, and the position information of the completely obscured objects is determined to facilitate users to quickly find invisible objects.

[0116] Optionally, in the step of determining location indication information of the object to be found based on the three-dimensional object map in response to the user's request to find the object to be found, after receiving the user's request to find the object to be found, the three-dimensional object map can be searched for a target object corresponding to the object to be found; in the three-dimensional object map, target location information corresponding to the target object can be determined; and based on the target location information, location indication information indicating the object to be found can be generated.

[0117] In the embodiment of the present invention, when a user issues a search request, a target item matching the item requested by the user can be searched in the constructed three-dimensional item map. Quick retrieval can be performed in the three-dimensional item map based on the item information provided by the user, such as the item name, category or feature.

[0118] After searching for the target item that matches the user's request, the specific location information of the target item can be further determined in the three-dimensional item map, and the location of the target item can be accurately located according to the spatial coordinates and item placement information in the three-dimensional item map. Based on the determined target item location information, location indication information indicating the item to be found is generated. The above-mentioned location indication information may include text descriptions, graphical displays, or virtual reality / augmented reality (VR / AR) navigation and other forms, so that users can intuitively understand and find the target item. For example, the system can generate a detailed path plan from the user's current location to the target item, or display an arrow pointing to the target item on the user's mobile device.

[0119] In a possible embodiment, when the number of the above-mentioned items to be found is one and the number of the above-mentioned target items is multiple, the multiple target items can be provided to the user for selection, so that the final item that the user wants to select can be determined more accurately. After the user selects the final item from the multiple target items, a detailed path planning from the user's current location to the final item is generated, thereby obtaining the location indication information of the item to be found. For example, if the user wants to take a piece of meat out of the refrigerator, and there are two pieces of meat in the refrigerator, the item information of the two pieces of meat can be provided to the user for selection, and the meat selected by the user is used as the final item.

[0120] In a possible embodiment, when there are multiple objects to be found and multiple target objects, and one object to be found corresponds to one target object, the order of picking up the multiple target objects can be planned based on the position indication information of the multiple target objects, and the position indication information indicating the multiple objects to be found can be generated based on the picking up order of the multiple target objects and the position indication information of the multiple target objects, so as to reduce the movement of the remaining objects. For example, the order of picking up the multiple target objects can be planned in at least one of the following order: from outside to inside, from top to bottom, and from small to large.

[0121] In a possible embodiment, when a user wears a bracelet / watch to look for items in a refrigerator, he or she can input the items to be found in the user interface by voice or text, and find the location of the target item in the three-dimensional item map in combination with the positioning of the bracelet / watch, and provide a search prompt through the bracelet / watch; when the positioning of the bracelet / watch deviates from the location of the item to be found, a voice reminder is given, such as a prompt that the apple is at 1 o'clock on your hand; when the positioning of the bracelet / watch coincides with the location of the item to be found, a prompt is given that the item can be taken. This can be applied to people who are not familiar with the placement of items in the refrigerator, elderly people with poor eyesight, or visually impaired people, improving the user experience.

[0122] like Figure 2 As shown, an embodiment of the present invention provides an object finding device, the object finding device comprising:

[0123] An acquisition module 201 is used to acquire initial image data of a target space and acquire an object database corresponding to the target space, wherein the object database includes object data corresponding to each object;

[0124] A construction module 202, configured to construct a three-dimensional object map of the target space based on the initial image data and the object database, wherein the three-dimensional object map includes location information of each of the objects;

[0125] The processing module 203 is used to respond to the user's request for finding an object to be found, and determine the location indication information of the object to be found based on the three-dimensional object map based on the location information of the object;

[0126] The sending module 204 is used to send the location indication information to the terminal device of the user, so that the terminal device displays the location indication information.

[0127] Optionally, the acquisition module 201 is also used to obtain the online shopping order information of the user; determine the first item data that needs to be placed in the target space based on the online shopping order information; obtain the second item data uploaded by the user; and construct an item database corresponding to the target space based on the first item data and the second item data.

[0128] Optionally, the construction module 202 is also used to perform feature extraction processing on the initial image data to obtain item features corresponding to each item; based on the item features, determine unobstructed items and obscured items in the item database; based on the item data corresponding to the unobstructed items and the item data corresponding to the obscured items, determine a first three-dimensional target state corresponding to the unobstructed items and a second three-dimensional target state corresponding to the obscured items; based on the first three-dimensional target state, the second three-dimensional target state and the initial image data, construct a three-dimensional object map of the target space.

[0129] Optionally, the construction module 202 is also used to perform three-dimensional reconstruction on the object data corresponding to the unobstructed object to obtain a first three-dimensional model corresponding to the unobstructed object; and to perform three-dimensional reconstruction on the object data corresponding to the obstructed object to obtain a second three-dimensional model corresponding to the obstructed object; based on the first three-dimensional model and the second three-dimensional model, obtain a first three-dimensional target state corresponding to the unobstructed object and a second three-dimensional target state corresponding to the obstructed object, respectively.

[0130] Optionally, the construction module 202 is also used to construct an initial three-dimensional object map of the target space based on the first three-dimensional target state and the initial image data, wherein the initial three-dimensional object map includes occupied space and unoccupied space, and the occupied space is occupied by the unobstructed objects; based on the second three-dimensional target state, the obstructed objects are spatially matched in the unoccupied space to obtain a three-dimensional object map of the target space.

[0131] Optionally, the construction module 202 is also used to determine the required space corresponding to the obstructed object based on the second three-dimensional target state; based on the required space corresponding to the obstructed object, spatially match the obstructed object in the unoccupied space to obtain a three-dimensional object map of the target space.

[0132] Optionally, the processing module 203 is also used to search for a target object corresponding to the object to be found in the three-dimensional object map after receiving a search request from a user for the object to be found; based on the location information of the object, determine the target location information corresponding to the target object in the three-dimensional object map; and based on the target location information, generate location indication information indicating the object to be found.

[0133] It should be noted that the object search device provided in the embodiment of the present invention can be applied to devices such as refrigerators, smart phones, tablets, computers, servers, etc. that can perform object search methods.

[0134] The object search device provided in the embodiment of the present invention can implement each process implemented by the object search method in the above method embodiment and can achieve the same beneficial effects. To avoid repetition, it will not be described here.

[0135] See also Figure 3 , Figure 3 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, such as Figure 3 As shown, it includes: a memory 302, a processor 301, and a computer program of an object search method stored in the memory 302 and executable on the processor 301, wherein:

[0136] The processor 301 is used to call the computer program stored in the memory 302 and execute the following steps:

[0137] Acquire initial image data of the target space, and acquire an object database corresponding to the target space, wherein the object database includes object data corresponding to each object;

[0138] Based on the initial image data and the object database, construct a three-dimensional object map of the target space, wherein the three-dimensional object map includes location information of each of the objects;

[0139] In response to a user's request for finding an object to be found, based on the location information of the object, determining location indication information of the object to be found on the basis of the three-dimensional object map;

[0140] The location indication information is sent to the terminal device of the user, so that the terminal device displays the location indication information.

[0141] Optionally, the step of acquiring initial object data placed in the target space performed by the processor 301 includes:

[0142] Obtaining the user's online shopping order information;

[0143] Determining first item data that needs to be placed in the target space according to the online shopping order information;

[0144] Acquire second item data uploaded by the user;

[0145] An object database corresponding to the target space is constructed based on the first object data and the second object data.

[0146] Optionally, the step of constructing the three-dimensional object map of the target space based on the initial image data and the object database performed by the processor 301 includes:

[0147] Performing feature extraction processing on the initial image data to obtain object features corresponding to each object;

[0148] Based on the object features, determining unobstructed objects and obstructed objects in the object database;

[0149] Determine, based on the object data corresponding to the unobstructed object and the object data corresponding to the obstructed object, a first three-dimensional target state corresponding to the unobstructed object and a second three-dimensional target state corresponding to the obstructed object;

[0150] A three-dimensional object map of the target space is constructed based on the first three-dimensional target state, the second three-dimensional target state and the initial image data.

[0151] Optionally, the step of determining, based on the object data corresponding to the unobstructed object and the object data corresponding to the obstructed object, a first three-dimensional target state corresponding to the unobstructed object and a second three-dimensional target state corresponding to the obstructed object, performed by the processor 301 includes:

[0152] Performing three-dimensional reconstruction on the object data corresponding to the unobstructed object to obtain a first three-dimensional model corresponding to the unobstructed object;

[0153] and, performing three-dimensional reconstruction on the object data corresponding to the obscured object to obtain a second three-dimensional model corresponding to the obscured object;

[0154] Based on the first three-dimensional model and the second three-dimensional model, a first three-dimensional target state corresponding to the unobstructed object and a second three-dimensional target state corresponding to the obstructed object are obtained respectively.

[0155] Optionally, the step of constructing the three-dimensional object map of the target space based on the first three-dimensional target state, the second three-dimensional target state and the initial image data, performed by the processor 301, includes:

[0156] Based on the first three-dimensional target state and the initial image data, construct an initial three-dimensional object map of the target space, wherein the initial three-dimensional object map includes occupied space and unoccupied space, and the occupied space is occupied by the unobstructed objects;

[0157] Based on the second three-dimensional target state, the blocked object is spatially matched in the unoccupied space to obtain a three-dimensional object map of the target space.

[0158] Optionally, the step of spatially matching the blocked object in the unoccupied space based on the second three-dimensional target state to obtain a three-dimensional object map of the target space, performed by the processor 301, includes:

[0159] Based on the second three-dimensional target state, determining a required space corresponding to the obstructed object;

[0160] Based on the required space corresponding to the obstructed object, the obstructed object is spatially matched in the unoccupied space to obtain a three-dimensional object map of the target space.

[0161] Optionally, the step of responding to a user's request to find an object to be found and determining, based on the location information of the object and on the three-dimensional object map, the location indication information of the object to be found, performed by the processor 301 includes:

[0162] After receiving a search request from a user for an object to be found, searching the three-dimensional object map for a target object corresponding to the object to be found;

[0163] Based on the location information of the object, determining the target location information corresponding to the target object in the three-dimensional object map;

[0164] Based on the target location information, location indication information indicating the object to be found is generated.

[0165] It should be noted that the electronic device provided by the embodiment of the present invention can be applied to devices such as smart refrigerators, smart phones, tablets, computers, servers, etc. that can perform object search methods.

[0166] The electronic device provided in the embodiment of the present invention can implement each process implemented by the object search method in the above method embodiment and can achieve the same beneficial effect. To avoid repetition, it will not be described here.

[0167] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the object searching method provided in the embodiment of the present invention are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0168] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The computer-readable storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM).

[0169] The above disclosure is only the preferred embodiment of the present invention, which certainly cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. A method for finding an object, characterized in that: The method comprises the following steps: Acquire initial image data of the target space, and acquire an object database corresponding to the target space, wherein the object database includes object data corresponding to each object; Based on the initial image data and the object database, construct a three-dimensional object map of the target space, wherein the three-dimensional object map includes location information of each of the objects; In response to a user's request for finding an object to be found, based on the location information of the object, determining location indication information of the object to be found on the basis of the three-dimensional object map; The location indication information is sent to the terminal device of the user, so that the terminal device displays the location indication information.

2. The object search method according to claim 1, characterized in that: The step of obtaining the object database corresponding to the target space includes: Obtaining the user's online shopping order information; Determining first item data that needs to be placed in the target space according to the online shopping order information; Acquire second item data uploaded by the user; An object database corresponding to the target space is constructed based on the first object data and the second object data.

3. The object search method according to claim 2, characterized in that: The step of constructing a three-dimensional object map of the target space based on the initial image data and the object database includes: Performing feature extraction processing on the initial image data to obtain object features corresponding to each object; Based on the object features, determining unobstructed objects and obstructed objects in the object database; Determine, based on the object data corresponding to the unobstructed object and the object data corresponding to the obstructed object, a first three-dimensional target state corresponding to the unobstructed object and a second three-dimensional target state corresponding to the obstructed object; A three-dimensional object map of the target space is constructed based on the first three-dimensional target state, the second three-dimensional target state and the initial image data.

4. The object search method according to claim 3, characterized in that: The step of determining a first three-dimensional target state corresponding to the unobstructed object and a second three-dimensional target state corresponding to the obstructed object based on the object data corresponding to the unobstructed object and the object data corresponding to the obstructed object comprises: Performing three-dimensional reconstruction on the object data corresponding to the unobstructed object to obtain a first three-dimensional model corresponding to the unobstructed object; and, performing three-dimensional reconstruction on the object data corresponding to the obscured object to obtain a second three-dimensional model corresponding to the obscured object; Based on the first three-dimensional model and the second three-dimensional model, a first three-dimensional target state corresponding to the unobstructed object and a second three-dimensional target state corresponding to the obstructed object are obtained respectively.

5. The object search method according to claim 3, characterized in that: The step of constructing the three-dimensional object map of the target space based on the first three-dimensional target state, the second three-dimensional target state and the initial image data comprises: Based on the first three-dimensional target state and the initial image data, construct an initial three-dimensional object map of the target space, wherein the initial three-dimensional object map includes occupied space and unoccupied space, and the occupied space is occupied by the unobstructed objects; Based on the second three-dimensional target state, the blocked object is spatially matched in the unoccupied space to obtain a three-dimensional object map of the target space.

6. The object search method according to claim 4, characterized in that: The step of spatially matching the blocked object in the unoccupied space based on the second three-dimensional target state to obtain a three-dimensional object map of the target space includes: Based on the second three-dimensional target state, determining a required space corresponding to the obstructed object; Based on the required space corresponding to the obstructed object, the obstructed object is spatially matched in the unoccupied space to obtain a three-dimensional object map of the target space.

7. The object searching method according to any one of claims 1 to 6, characterized in that: The step of responding to a user's request for finding an object to be found and determining the location indication information of the object to be found based on the three-dimensional object map based on the location information of the object comprises: After receiving a search request from a user for an object to be found, searching the three-dimensional object map for a target object corresponding to the object to be found; Based on the location information of the object, determining the target location information corresponding to the target object in the three-dimensional object map; Based on the target location information, location indication information indicating the object to be found is generated.

8. An object finding device, characterized in that: The object finding device comprises: An acquisition module, used to acquire initial image data of a target space, and acquire an object database corresponding to the target space, wherein the object database includes object data corresponding to each object; A construction module, configured to construct a three-dimensional object map of the target space based on the initial image data and the object database, wherein the three-dimensional object map includes location information of each of the objects; A processing module, for responding to a user's request for finding an object to be found, and determining, based on the location information of the object and on the three-dimensional object map, location indication information of the object to be found; The sending module is used to send the location indication information to the terminal device of the user, so that the terminal device displays the location indication information.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the object search method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the object search method according to any one of claims 1 to 7 are implemented.

Images