Method, device, electronic device and medium for storing item location information

Abstract

The embodiments of the present disclosure disclose a method, apparatus, electronic device, and medium for storing object location information. A specific implementation of the method includes: acquiring target depth data, wherein the target depth data is obtained based on depth data captured for the area where the object to be identified is located; identifying the object to be identified based on the target depth data; generating object location information of the object to be identified based on the relative relationship between the position of the device capturing the depth data and the position of the identified object to be identified, wherein the object location information includes altitude information; and associating and storing the object location information of the object to be identified with a matching object tag. This implementation realizes the automatic recording of multi-dimensional location information of an object, providing a solid data foundation for subsequent object searches.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and more particularly, to a method, apparatus, electronic device, and medium for storing object location information. Background Art

[0002] In real-life scenarios, it is often necessary to record the locations of various objects (such as glasses on a table, vehicles in a parking lot, etc.) for later retrieval.

[0003] Existing technology typically involves users using mobile phones or other devices to record the location of items in writing or taking photos. However, due to variations in location descriptions, the desired item is often not accurately located. Existing technology also involves users using positioning devices in mobile phones or other devices to locate the location of items and save the location information. However, positioning accuracy in indoor environments is poor and typically only records two-dimensional locations. The lack of spatial information often makes it impossible to accurately record the location. Summary of the Invention

[0004] Embodiments of the present disclosure provide a method, apparatus, electronic device, and medium for storing item location information.

[0005] In a first aspect, an embodiment of the present disclosure provides a method for storing object location information, the method comprising: acquiring target depth data, wherein the target depth data is obtained based on depth data captured for an area where an object to be identified is located; identifying the object to be identified based on the target depth data; generating object location information of the object to be identified based on a relative relationship between a position of a device capturing the depth data and the position of the identified object to be identified, wherein the object location information includes altitude information; and storing the object location information of the object to be identified in association with a matching object tag.

[0006] In a second aspect, an embodiment of the present disclosure provides a method for displaying item search indication information, the method comprising: in response to receiving item search information, selecting item location information that matches the item tag indicated by the item search information from a preset item database, wherein the item database includes item location information and corresponding item tags; in response to detecting that a camera is turned on, obtaining the location information of the camera; generating item search indication information based on the distance between the location indicated by the camera location information and the location indicated by the selected item location information; and displaying the item search indication information on an image captured by the camera.

[0007] In a third aspect, an embodiment of the present disclosure provides a device for storing object location information, the device comprising: a first acquisition unit, configured to acquire target depth data, wherein the target depth data is obtained based on depth data captured for an area where the object to be identified is located; an identification unit, configured to identify the object to be identified based on the target depth data; a first generation unit, configured to generate object location information of the object to be identified based on a relative relationship between a position of a shooting device of the depth data and the position of the identified object to be identified, wherein the object location information includes altitude information; an association unit, configured to store the object location information of the object to be identified in association with a matching object tag.

[0008] In a fourth aspect, an embodiment of the present disclosure provides a device for displaying item search indication information, the device comprising: a selection unit, configured to, in response to receiving item search information, select item location information that matches the item tag indicated by the item search information from a preset item database, wherein the item database includes item location information and corresponding item tags; a second acquisition unit, configured to, in response to detecting that the camera is turned on, acquire the location information of the camera; a second generation unit, configured to generate item search indication information based on the distance between the location indicated by the camera location information and the location indicated by the selected item location information; and a display unit, configured to display the item search indication information on an image captured by the camera.

[0009] In a fifth aspect, an embodiment of the present disclosure provides an electronic device comprising: one or more processors; a storage device on which one or more programs are stored; when the one or more programs are executed by one or more processors, the one or more processors implement the method described in any implementation method in the first aspect.

[0010] In a sixth aspect, an embodiment of the present disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0011] The embodiments of the present disclosure provide methods, devices, electronic devices, and media for storing item location information. By using the distance indicated by the acquired depth data and the position between the camera and the location device to generate item location information including altitude information, the automatic recording of multi-dimensional location information of the item is achieved, providing a solid data foundation for subsequent item searches. The embodiments of the present disclosure also provide methods, devices, electronic devices, and media for displaying item search indication information. Location information matching the item to be searched is selected through a preset item database, and the generated item search indication information can be displayed on the image captured by the camera. This avoids the need for users to switch visually between the surrounding environment and the screen displaying the indication information when searching for items using the terminal. The item search indication information can be displayed intuitively, improving the efficiency of item searches. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Other features, objects and advantages of the present disclosure will become more apparent from a reading of the detailed description of non-limiting embodiments made with reference to the following drawings:

[0013] Figure 1 is an exemplary system architecture diagram in which an embodiment of the present disclosure may be applied;

[0014] Figure 2 is a flow chart of an embodiment of a method for storing item location information according to the present disclosure;

[0015] Figure 3 is a schematic diagram of an application scenario of a method for storing item location information according to an embodiment of the present disclosure;

[0016] Figure 4 is a flow chart of an embodiment of a method for displaying item search indication information according to the present disclosure;

[0017] Figure 5 is a schematic structural diagram of an embodiment of a device for storing item location information according to the present disclosure;

[0018] Figure 6 is a structural diagram of an embodiment of a device for displaying item search indication information according to the present disclosure;

[0019] Figure 7 It is a schematic structural diagram of an electronic device suitable for implementing the embodiments of the present disclosure. DETAILED DESCRIPTION

[0020] The present disclosure will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are intended only to illustrate the relevant invention and are not intended to limit the invention. It should also be noted that, for ease of description, only portions relevant to the relevant invention are shown in the accompanying drawings.

[0021] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in the present disclosure can be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0022] Figure 1 An exemplary architecture 100 is shown to which the method for storing item location information or the apparatus for storing item location information of the present disclosure can be applied.

[0023] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. Network 104 is a medium for providing communication links between terminal devices 101, 102, 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0024] The terminal devices 101, 102, and 103 interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications may be installed on the terminal devices 101, 102, and 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, image recognition applications, database applications, etc.

[0025] The terminal devices 101, 102, and 103 can be hardware or software. When the terminal devices 101, 102, and 103 are hardware, they can be various electronic devices with display screens and supporting human-computer interaction, including but not limited to smartphones, tablet computers, laptop computers, and desktop computers. When the terminal devices 101, 102, and 103 are software, they can be installed in the electronic devices listed above. They can be implemented as multiple software or software modules (for example, software or software modules for providing distributed services), or they can be implemented as a single software or software module. No specific limitation is made here.

[0026] Server 105 can be a server that provides various services, such as a backend server that supports image recognition applications and database applications on terminal devices 101, 102, and 103. The backend server can process the received target depth data and associate and store the identified objects based on the target depth data with the matching object tags. Alternatively, the backend server can send the generated object location information to terminal devices 101, 102, and 103, so that terminal devices 101, 102, and 103 can associate and store the object location information with the matching object tags.

[0027] It should be noted that the server can be either hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software or software modules (e.g., software or software modules for providing distributed services), or as a single software or software module. No specific limitations are given here.

[0028] It should be noted that the method for storing item location information provided in the embodiments of the present disclosure is generally executed by the server 105, and accordingly, the device for storing item location information is generally provided in the server 105. Optionally, under the condition of sufficient computing power, the method for storing item location information provided in the embodiments of the present application may also be executed by the terminal devices 101, 102, and 103, and accordingly, the device for storing item location information may also be provided in the terminal devices 101, 102, and 103. Optionally, the method for storing item location information provided in the embodiments of the present application may also be performed by the server 105 and the terminal devices 101, 102, and 103, respectively, with no limitation herein.

[0029] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0030] Continue to refer Figure 2 , shows a process 200 of an embodiment of a method for storing item location information according to the present disclosure. The method for storing item location information includes the following steps:

[0031] Step 201: Acquire target depth data.

[0032] In this embodiment, the execution subject of the method for storing item location information (such as Figure 1The terminal devices 101, 102, and 103 shown in the figure can obtain target depth data through a wired connection or a wireless connection. The target depth data is obtained based on the depth data captured for the area where the object to be identified is located. The depth data may include a depth image. Each pixel in the depth image can be used to characterize the distance (depth) from the image collector to each point in the scene, which can directly reflect the geometric shape of the visible surface of the object. The depth data can be obtained in various ways, such as lidar depth imaging, computer stereo vision imaging, coordinate measuring machine method, moiré fringe method, structured light method, etc.

[0033] As an example, the execution subject may obtain target depth data pre-stored locally, or may obtain the target depth data from an electronic device (e.g., a lidar scanner) to which it is in communication. The target depth data may be depth data captured for the area where the object to be identified is located, or may be data obtained by pre-processing the depth data captured for the area where the object to be identified is located.

[0034] In some optional implementations of this embodiment, the above-mentioned depth data is also associated with corresponding confidence data. The above-mentioned confidence data can be used to characterize the credibility of each pixel value in the depth image. As an example, the above-mentioned depth data and the associated corresponding confidence data may include AVDepthData of the iOS platform. The data in the above-mentioned AVDepthData may include depth data (such as depthMap) obtained based on the lidar scanner in the iOS device and its corresponding confidence data (such as confidenceMap).

[0035] Based on the above optional implementation, the above execution subject can also obtain the target depth data according to the following steps:

[0036] The first step is to obtain depth data of the area where the object to be identified is located.

[0037] In these implementations, the execution entity may obtain depth data captured for the area where the object to be identified is located in a manner consistent with the above.

[0038] In the second step, the depth data with a corresponding confidence level not less than a preset threshold is extracted from the acquired depth data to generate target depth data.

[0039] In these implementations, the execution entity may extract corresponding depth data having a confidence level not less than a preset threshold from the depth data acquired in the first step to generate target depth data.

[0040] Based on the above optional implementation methods, this solution can reduce the adverse effects of low-confidence depth data on the accuracy of object recognition results, and help to further improve the accuracy of the generated object location information.

[0041] Step 202: Identify the object to be identified based on the target depth data.

[0042] In this embodiment, based on the target depth data acquired in step 201 , the execution entity may identify the object to be identified in various ways.

[0043] As an example, the execution entity may first transform the target depth data acquired in step 201 into point cloud data through coordinate conversion. Subsequently, the execution entity may utilize various point cloud target detection methods to perform target detection on the generated point cloud data to identify the object to be identified. Thus, based on the identification results of the object to be identified, the execution entity can typically determine the relative positional relationship between the object to be identified and the device capturing the depth data.

[0044] In some optional implementations of this embodiment, the execution entity may input the target depth data acquired in step 201 into a pre-trained object recognition model to generate location indication information indicating the location of the object to be recognized, so as to recognize the object to be recognized.

[0045] In these implementations, the object recognition model may include various target detection models trained using machine learning methods. The position indication information may include various forms. As an example, the position indication information may be used to indicate the position of the identified object to be identified in the depth map corresponding to the target depth data. The position indication information may be, for example, a detection box or the relative positional relationship between the geometric center of the identified object to be identified and the device capturing the depth data.

[0046] Therefore, this solution can use the model trained by machine learning to identify and locate objects based on depth images.

[0047] Based on the above optional implementation, the above object recognition model can be trained through the following steps:

[0048] S1. Obtain a training sample set.

[0049] In these implementations, the execution entity used to train the object recognition model can obtain a training sample set through various methods. The training sample set may include sample depth data and sample annotation information. The sample annotation information may be used to indicate the location of the sample object corresponding to the sample depth data. As an example, after obtaining the sample depth data, the execution entity may also obtain the depth data corresponding to the sample annotation information. The sample annotation information may be used to indicate the portion of the sample depth data corresponding to the area of ​​the object to be recognized.

[0050] Optionally, the execution entity may further normalize the depth data corresponding to the sample annotation information to generate a matrix of dimensions [m, n, h]. Here, m, n, and h may respectively represent the maximum length and maximum width of the area corresponding to the object to be identified in the sample depth data, and the difference between the depth data corresponding to the area to be identified and the maximum depth in the target depth data. Optionally, the execution entity may further flatten the matrix of dimensions [m, n, h] into a matrix of dimensions [m×n×h, 1].

[0051] Based on the above optional implementation methods, this solution can use data normalization to improve the speed and effect of model training.

[0052] Optionally, the above sample annotation information can also be used to indicate the category of the object to be identified, such as vehicles, glasses, etc.

[0053] S2. Take the sample depth data of the training samples in the training sample set as input, take the sample annotation information corresponding to the input sample depth data as the expected output, and train to obtain an object recognition model.

[0054] In these implementations, the execution entity may train an initial object recognition model using a supervised learning approach. When a training stop condition is met, the execution entity may determine the trained model as the object recognition model. The training stop condition may, for example, include verification through precision-recall (PR).

[0055] Optionally, the above-mentioned object recognition model may also output the category of the object to be recognized.

[0056] Based on the above optional implementation method, this solution can utilize sample annotation information to indicate the location of the sample object corresponding to the sample depth data, and adopt a supervised method to train the object recognition model. Compared with the existing two-dimensional object image recognition, it can effectively determine the relative position of the object relative to the shooting device of the depth data.

[0057] Step 203 : generating object position information of the object to be identified based on the relative relationship between the position of the photographing device of the depth data and the identified position of the object to be identified.

[0058] In this embodiment, based on the relative relationship between the position of the camera capturing the depth data and the position of the identified object, the execution entity may generate the object location information of the object in various ways, wherein the object location information includes altitude information.

[0059] As an example, based on the depth data corresponding to the area where the object to be identified is located identified in the above step 202, the above execution entity can determine the depth value corresponding to the feature point of the above object to be identified in various ways. For example, the above execution entity can perform sparse sampling on the depth data corresponding to the area where the object to be identified is located, and calculate the average value of the depth data obtained by sampling. The above average value may include but is not limited to at least one of the following: arithmetic mean, geometric mean, weighted mean. Correspondingly, the above feature point can be the center of the surface of the object, the geometric center, etc. Then, the above execution entity can obtain the shooting device of the depth data (for example Figure 1 The above-mentioned location can be represented by, for example, longitude, latitude, and altitude. Optionally, the above-mentioned location can also include the attitude angle of the shooting device of the above-mentioned depth data. Then, based on the acquired attitude angle and the depth value corresponding to the feature point of the above-mentioned object to be identified, the above-mentioned execution entity can determine the longitude difference, latitude difference, and altitude difference between the above-mentioned object to be identified and the above-mentioned shooting device. Thus, based on the longitude, latitude, and altitude of the above-mentioned shooting device and the corresponding longitude difference, latitude difference, and altitude difference, the above-mentioned execution entity can generate the object location information of the above-mentioned object to be identified.

[0060] In some optional implementations of this embodiment, based on the relative relationship between the position of the camera device in the depth data and the position of the identified object to be identified, the execution entity may generate the position information of the object to be identified according to the following steps:

[0061] The first step is to present the location of the identified object to be identified.

[0062] In these implementations, the execution entity may present the location of the identified object in step 202 in various ways. For example, the execution entity may display a frame indicating the location of the identified object on a screen. As another example, the execution entity may transmit location information representing the location of the identified object to a communication-connected display device.

[0063] In the second step, in response to receiving confirmation information from the user regarding the location of the presented object to be identified, extracting depth data of the object to be identified indicated by the confirmation information.

[0064] In these implementations, in response to receiving confirmation information from the user regarding the location of the presented object to be identified, the execution entity may extract depth data for the object to be identified indicated by the confirmation information. As an example, in response to determining that the confirmation information indicates that the presented location of the object to be identified is correct, the execution entity may extract depth data corresponding to the area where the identified object to be identified is located. As another example, in response to determining that the confirmation information indicates a user-corrected location of the object to be identified, the execution entity may extract depth data corresponding to the corrected location.

[0065] The third step is to generate a position offset based on the extracted depth data.

[0066] In these implementations, the execution entity may generate the position offset in a manner consistent with the aforementioned steps. The form of the position offset is generally consistent with the position of the camera capturing the depth data.

[0067] The fourth step is to adjust the position of the camera for the depth data based on the position offset to generate the position information of the object to be identified.

[0068] In these implementations, as an example, the execution entity may add the position offset generated in the third step to the corresponding position information of the shooting device for indicating depth data to generate the position information of the object to be identified.

[0069] Based on the above optional implementation, this solution can extract the depth data of the object to be identified based on the received user confirmation information, thereby helping to improve the accuracy of the generated location information of the object to be identified.

[0070] Step 204: Store the item location information of the item to be identified in association with the matching item tag.

[0071] In this embodiment, the above-mentioned execution entity can associate the item location information of the item to be identified identified in the above-mentioned step 202 with the matching item tag in various ways. Among them, the above-mentioned matching item tag can be used to identify the above-mentioned item to be identified, which can be obtained in various ways. As an example, the above-mentioned item tag can be input or selected by the user, such as "my mobile phone". As another example, the above-mentioned item tag can also be generated according to the item category output by the above-mentioned item recognition model, such as "vehicle 20201202123618", which can be used to represent the vehicle photographed at 12:36:18 on December 2, 2020. Afterwards, the above-mentioned execution entity can associate and store the item location information of the above-mentioned item to be identified with the matching item tag.

[0072] Continue to see Figure 3 , Figure 3 FIG. 1 is a schematic diagram of an application scenario of a method for storing item location information according to an embodiment of the present disclosure. Figure 3 In an application scenario, user 301 uses a lidar scanner installed on terminal device 302 to capture the area where a table is located, generating a depth map 303. Based on depth map 303, terminal device 302 identifies table 304. Based on the relative positional relationship between terminal device 302's location 305 and the identified table 304, item location information 306 for the table is generated. Terminal device 302 then associates and stores item location information 306 with the corresponding item tag "table" as a piece of stored data 307.

[0073] Currently, one existing technology typically uses text or photography to record item information, resulting in inaccurate positioning. Another existing technology uses methods that locate the item's placement and store the location information. However, because the location information often lacks altitude data, the stored location information is incomplete. The method provided by the above-mentioned embodiments of the present disclosure generates item location information including altitude information by using the distance indicated by the depth data and the position of the camera. This achieves automatic recording of the item's multi-dimensional location information, providing a solid data foundation for subsequent item searches.

[0074] Further references Figure 4 , which shows a process 400 of another embodiment of a method for displaying item search indication information. The process 400 of the method for displaying item search indication information includes the following steps:

[0075] Step 401: In response to receiving item search information, select item location information that matches the item tag indicated by the item search information from a preset item database.

[0076] In this embodiment, in response to receiving the item search information, the execution subject (eg Figure 1 The terminal devices 101, 102, and 103 shown in the figure can select item location information that matches the item tag indicated by the item search information from a preset item database in various ways. The item search information can be used to indicate the search for the item to be found. The item database can include item location information and corresponding item tags.

[0077] In this embodiment, as an example, a user can enter a tag for the item to be searched, such as "My Car," on a page displayed on a terminal device (the aforementioned execution entity). The user can then click the "Start Search" button on the page. Consequently, the execution entity can receive the item search information indicating a search for My Car. The execution entity can then select item location information from a pre-set item database that matches the item tag indicated in the item search information. This item location information could, for example, be "B128, Basement 1." Alternatively, the item location information could be recorded latitude and longitude information.

[0078] In some optional implementations of this embodiment, the above-mentioned item location information may further include altitude information, wherein the above-mentioned altitude information may be altitude, or other information associated with altitude that represents height in three-dimensional space, which is not limited here.

[0079] In some optional implementations of this embodiment, the aforementioned preset item database may be obtained based on the method for storing item location information described in the aforementioned embodiment.

[0080] Step 402: In response to detecting that the camera is turned on, obtain the location information of the camera.

[0081] In this embodiment, in response to detecting that the camera is turned on, the execution entity may obtain the camera's location information through various means. As an example, the execution entity may obtain the camera's location information from a positioning device. The positioning device is typically installed on the same electronic device as the camera, such as a mobile phone.

[0082] Optionally, the above-mentioned position information may also include an attitude angle.

[0083] Step 403 : Generate object search indication information based on the distance between the position indicated by the camera position information and the position indicated by the selected object position information.

[0084] In this embodiment, based on the distance between the location indicated by the camera location information acquired in step 402 and the location indicated by the item location information selected in step 401, the execution entity may generate item search indication information in various ways. For example, in response to determining that the distance is greater than a preset distance threshold, the execution entity may generate item search indication information indicating that the item is not nearby. Alternatively, the execution entity may also generate item search indication information directed to the location indicated by the item location information.

[0085] In some optional implementations of this embodiment, based on the optional implementation of the above-mentioned step 402, in response to determining that the altitude between the altitude indicated by the camera's position information and the altitude indicated by the selected object's position information is greater than a preset altitude threshold, the above-mentioned execution entity can generate augmented reality prompt information for indicating that the object to be found is not on the current horizontal plane.

[0086] Based on the above optional implementation, this solution can prompt in time when the object to be found is not within the current altitude range, so as to reduce the waste of search time.

[0087] In some optional implementations of this embodiment, based on the optional implementation of step 402, the execution entity may generate the item search indication information based on the distance between the position indicated by the camera position information and the position indicated by the selected item position information by performing the following steps:

[0088] In a first step, in response to determining that the altitude difference between the altitude indicated by the camera position information and the location indicated by the selected object location information is not greater than a preset altitude threshold, a depth image matching the image captured by the camera is obtained.

[0089] In these implementations, in response to determining that the altitude between the altitude indicated by the camera's location information and the altitude indicated by the selected object's location information is not greater than a preset altitude threshold, the execution entity may obtain a depth image that matches the image captured by the camera in various ways. As an example, the execution entity (such as a mobile phone) may be equipped with an optical camera and a depth camera. The shooting directions of the optical camera and the depth camera are generally the same. Thus, when a user uses a mobile phone, the execution entity may obtain a depth image captured by the optical camera through the depth camera. The depth camera may also be other devices for acquiring depth images, such as a lidar scanner.

[0090] The second step is to determine the position of the object indicated by the depth data in the depth image based on the position information of the camera.

[0091] In these implementations, based on the position information of the camera acquired in step 402 , the execution entity may determine the position of the object indicated by the depth data in the depth image in various ways.

[0092] It should be noted that the specific implementation of the second step can refer to step 203 and its optional implementation in the above embodiment, and will not be repeated here.

[0093] In a third step, in response to determining that the position of the object indicated by the depth data in the depth image matches the selected object position information, augmented reality indication information for indicating the object to be found is generated.

[0094] In these implementations, in response to determining that the position of the object indicated by the depth data in the depth image matches the selected object location information, the execution entity may generate augmented reality indication information indicating the object to be found in various ways. Whether the position of the object indicated by the depth data in the depth image matches the selected object location information may be determined, for example, by whether the distance between the positions is less than a preset distance threshold (e.g., 2 meters). As an example, the augmented reality indication information may be, for example, "You have arrived near the object to be found" displayed using augmented reality technology.

[0095] Based on the above optional implementation methods, this solution can use enhanced display technology to provide intuitive prompts for the search route in the process of searching for the item to be found, thereby improving the efficiency of item search.

[0096] Step 404: Display item search indication information on the image captured by the camera.

[0097] In this embodiment, the execution entity can display the item search indication information on the image captured by the camera in various ways. For example, when searching for an item, a user typically uses the camera on a mobile phone (the execution entity) to capture the surrounding environment. Therefore, the execution entity can display the item search information generated in step 403 on the captured image displayed on the display screen.

[0098] In some optional implementations of this embodiment, based on the optional implementation of step 403, the execution entity may display the item search indication information on the image captured by the camera according to the following steps:

[0099] The first step is to obtain the display position of the object that matches the selected object position information on the image captured by the camera that matches the depth image.

[0100] In these implementations, the display position may include a specific pixel position or a preset range (eg, the upper left corner, the lower right corner, the middle position, etc.).

[0101] The second step is to determine the display position of the augmented reality indication information based on the display position.

[0102] In these implementations, based on the display position determined in the first step, the execution entity may determine the display position of the augmented reality indication information in various ways. As an example, the execution entity may first determine the distance between the display position and the edge of the image. In response to determining that the distance is greater than a preset display distance, the execution entity may add a preset offset to the corresponding value in the display position information corresponding to the display position. For example, the execution entity may add a preset small value to the vertical coordinate corresponding to the display position to determine that the augmented reality indication information is displayed above the display position.

[0103] The third step is to display item search instructions on the image captured by the camera according to the display location.

[0104] Based on the above optional implementation, this solution can use enhanced display technology to intuitively display item search indication information.

[0105] from Figure 4 As can be seen, process 400 of the method for displaying item search instructions in this embodiment selects location information matching the item to be found from a preset item database and displays the generated item search instructions on an image captured by a camera. This eliminates the need for users to switch between their surroundings and the screen displaying the instructions when searching for items using a terminal. This allows for intuitive display of the item search instructions, improving item search efficiency.

[0106] Further references Figure 5 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a device for storing item location information. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0107] like Figure 5As shown, the apparatus 500 for storing object location information provided in this embodiment includes a first acquisition unit 501, an identification unit 502, a first generation unit 503, and an association unit 504. The first acquisition unit 501 is configured to acquire target depth data, wherein the target depth data is obtained based on depth data captured for an area where the object to be identified is located; the identification unit 502 is configured to identify the object to be identified based on the target depth data; the first generation unit 503 is configured to generate object location information of the object to be identified based on the relative relationship between the position of the device capturing the depth data and the identified position of the object to be identified, wherein the object location information includes altitude information; and the association unit 504 is configured to associate and store the object location information of the object to be identified with a matching object tag.

[0108] In this embodiment, in the device 500 for storing item location information, the specific processing of the first acquisition unit 501, the identification unit 502, the first generation unit 503 and the association unit 504 and the technical effects thereof can be referred to in the respective Figure 2 The relevant descriptions of step 201, step 202, step 203 and step 204 in the corresponding embodiment are not repeated here.

[0109] In some optional implementations of this embodiment, the identification unit 502 may be further configured to: input the target depth data into a pre-trained object recognition model to generate position indication information for indicating the position of the object to be identified.

[0110] In some optional implementations of this embodiment, the above-mentioned object recognition model can be trained through the following steps: obtaining a training sample set, wherein the training sample set includes sample depth data and sample annotation information, and the sample annotation information is used to indicate the location of the sample object corresponding to the sample depth data; taking the sample depth data of the training samples in the training sample set as input, and taking the sample annotation information corresponding to the input sample depth data as the expected output, to train and obtain the object recognition model.

[0111] In some optional implementations of this embodiment, the above-mentioned first generation unit 503 can be further configured to: present the identified location of the object to be identified; in response to receiving confirmation information from the user regarding the presented location of the object to be identified, extract the depth data of the object to be identified indicated by the confirmation information; based on the extracted depth data, generate a position offset; based on the position offset, adjust the position of the shooting device of the depth data to generate the position information of the object to be identified.

[0112] In some optional implementations of this embodiment, the depth data may also be associated with corresponding confidence data. The first acquisition unit 501 may be further configured to: acquire depth data captured for the area where the object to be identified is located; and extract depth data with a confidence level not less than a preset threshold from the acquired depth data to generate target depth data.

[0113] The apparatus provided by the above-mentioned embodiments of the present disclosure generates object location information including altitude information based on the distance indicated by the depth data acquired by the first acquisition unit 501 and the position between the shooting device through the first generation unit 503, thereby realizing automatic recording of the multi-dimensional location information of the object and providing a solid data foundation for subsequent object searches.

[0114] Further references Figure 6 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a device for displaying item search indication information. Figure 4 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0115] like Figure 6 As shown, the apparatus 600 for displaying item search indication information provided in this embodiment includes a selection unit 601, a second acquisition unit 602, a second generation unit 603, and a display unit 604. The selection unit 601 is configured to, in response to receiving item search information, select item location information that matches the item tag indicated by the item search information from a preset item database, wherein the item database includes item location information and corresponding item tags; the second acquisition unit 602 is configured to, in response to detecting that a camera is turned on, acquire the camera's location information; the second generation unit 603 is configured to generate item search indication information based on the distance between the position indicated by the camera's location information and the position indicated by the selected item location information; and the display unit 604 is configured to display the item search indication information on an image captured by the camera.

[0116] In this embodiment, in the device for displaying item search indication information 600, the specific processing of the selection unit 601, the second acquisition unit 602, the second generation unit 603 and the display unit 604 and the technical effects thereof can be referred to respectively. Figure 4 The relevant descriptions of step 401, step 402, step 403 and step 404 in the corresponding embodiment are not repeated here.

[0117] In some optional implementations of this embodiment, the above-mentioned object location information may further include altitude information.

[0118] In some optional implementations of this embodiment, the above-mentioned second generation unit 603 can be further configured to: in response to determining that the altitude between the altitude indicated by the camera's position information and the altitude indicated by the selected object position information is greater than a preset altitude threshold, generate augmented reality prompt information for indicating that the object to be found is not on the current horizontal plane.

[0119] In some optional implementations of this embodiment, the above-mentioned second generation unit 603 can be further configured to: in response to determining that the altitude between the altitude indicated by the camera's position information and the position indicated by the selected object position information is not greater than a preset altitude threshold, obtain a depth image that matches the image taken by the camera; based on the camera's position information, determine the position of the object indicated by the depth data in the depth image; in response to determining that the position of the object indicated by the depth data in the depth image matches the selected object position information, generate augmented reality indication information for indicating the object to be found.

[0120] In some optional implementations of this embodiment, the above-mentioned display unit 604 can be further configured to: obtain the display position of the object matching the selected object location information on the image captured by the camera matching the depth image; determine the display position of the augmented reality indication information based on the display position; and display the object search indication information on the image captured by the camera based on the display position.

[0121] In some optional implementations of this embodiment, the above-mentioned preset item database can be based on the above-mentioned Figure 2 The method described in the embodiment is obtained.

[0122] In the apparatus provided by the above-described embodiment of the present disclosure, the selection unit 601 selects location information matching the item to be found from a preset item database, and the display unit 604 can display the item search indication information generated by the second generation unit 603 on the image captured by the camera. This eliminates the need for users to visually switch between the surrounding environment and the screen displaying the indication information when searching for items using the terminal. The item search indication information can be displayed intuitively, improving the efficiency of item search.

[0123] Reference below Figure 7 , which shows an electronic device (eg, Figure 1 The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, notebook computers, etc. and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The server shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0124] like Figure 7 As shown, the electronic device 700 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. Various programs and data required for the operation of the electronic device 700 are also stored in the RAM 703. The processing device 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0125] Typically, the following devices may be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 709. The communication device 709 may allow the electronic device 700 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7 The electronic device 700 is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead. Figure 7 Each block shown in the figure may represent one device, or may represent multiple devices as needed.

[0126] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network via the communication device 709, or installed from the storage device 708, or installed from the ROM 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0127] It should be noted that the computer-readable medium described in the embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or device. In the embodiments of the present disclosure, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or convey a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code embodied on the computer-readable medium may be conveyed using any suitable medium, including but not limited to wires, optical cables, RF (Radio Frequency), or any suitable combination thereof.

[0128] The computer-readable medium may be included in the electronic device, or may exist independently without being installed in the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains target depth data, wherein the target depth data is obtained based on depth data captured for the area where the object to be identified is located; identifies the object to be identified based on the target depth data; generates object location information of the object to be identified based on the relative relationship between the position of the device capturing the depth data and the position of the identified object to be identified, wherein the object location information includes altitude information; associates and stores the object location information of the object to be identified with the matching object tag; or

[0129] The electronic device is configured to: in response to receiving item search information, select item location information that matches the item tag indicated by the item search information from a preset item database, wherein the item database includes item location information and corresponding item tags; in response to detecting that a camera is turned on, obtain the location information of the camera; generate item search indication information based on the distance between the location indicated by the camera location information and the location indicated by the selected item location information; and display the item search indication information on an image captured by the camera.

[0130] Computer program code for performing the operations of embodiments of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0131] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to the various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0132] The units involved in the embodiments described in the present disclosure may be implemented by software or by hardware. The units described may also be provided in a processor, for example, they may be described as: a processor comprising a first acquisition unit, an identification unit, a first generation unit, and an association unit; or, a processor comprising a selection unit, a second acquisition unit, a second generation unit, and a display unit. The names of these units do not, in some cases, constitute limitations on the units themselves. For example, the first acquisition unit may also be described as a "unit for acquiring target depth data, wherein the target depth data is obtained based on depth data captured for the area where the object to be identified is located."

[0133] The above description is merely a preferred embodiment of the present disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also encompass other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned inventive concept. For example, a technical solution formed by mutually replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A method for storing item location information, comprising: Acquiring target depth data, wherein the target depth data is obtained based on depth data captured for an area where the object to be identified is located; Identifying the object to be identified based on the target depth data; generating, based on a relative relationship between a position of a capturing device according to the depth data and a position of the identified object to be identified, object location information of the object to be identified, wherein the object location information includes altitude information for displaying object search indication information, and the search indication information includes prompt information for indicating that the object to be identified is not on a current horizontal plane; The object location information of the object to be identified is associated with the matching object tag and stored.

2. The method according to claim 1, wherein The identifying the object to be identified based on the target depth data includes: The target depth data is input into a pre-trained object recognition model to generate position indication information for indicating the position of the object to be recognized.

3. The method according to claim 2, wherein: The object recognition model is trained by the following steps: Acquire a training sample set, wherein the training sample set includes sample depth data and sample annotation information, and the sample annotation information is used to indicate the location of the sample object corresponding to the sample depth data; The object recognition model is obtained by training by taking the sample depth data of the training samples in the training sample set as input and the sample annotation information corresponding to the input sample depth data as the expected output.

4. The method according to claim 1, wherein Generating the position information of the object to be identified based on the relative relationship between the position of the shooting device and the identified position of the object to be identified based on the depth data includes: Presenting the location of the identified object to be identified; In response to receiving confirmation information from the user regarding the location of the presented object to be identified, extracting depth data of the object to be identified indicated by the confirmation information; generating a position offset based on the extracted depth data; The position of the shooting device of the depth data is adjusted based on the position offset to generate the position information of the object to be identified.

5. The method according to any one of claims 1 to 4, wherein: The depth data is also associated with corresponding confidence data; and The acquiring target depth data includes: Obtain depth data captured for the area where the object to be identified is located; Depth data with a corresponding confidence level not less than a preset threshold is extracted from the acquired depth data to generate target depth data.

6. A method for displaying item search indication information, comprising: In response to receiving the item search information, selecting, from a preset item database, item location information that matches the item tag indicated by the item search information, wherein the item database includes item location information and corresponding item tags, and the item location information includes altitude information; In response to detecting that a camera is turned on, obtaining position information of the camera; Generating item search indication information based on a distance between a position indicated by the camera position information and a position indicated by the selected item position information, including: in response to determining that the altitude difference between the altitude indicated by the camera position information and the position indicated by the selected item position information is greater than a preset altitude threshold, generating augmented reality prompt information indicating that the item to be found is not at a current horizontal plane; The item search indication information is displayed on the image captured by the camera.

7. The method according to claim 6, wherein: The generating of the item search indication information based on the distance between the position indicated by the position information of the camera and the position indicated by the selected item position information includes: In response to determining that the altitude between the altitude indicated by the camera's location information and the location indicated by the selected object's location information is not greater than a preset altitude threshold, acquiring a depth image that matches the image captured by the camera; determining, based on the position information of the camera, a position of an object indicated by the depth data in the depth image; In response to determining that the position of the object indicated by the depth data in the depth image matches the selected object position information, augmented reality indication information for indicating the object to be found is generated.

8. The method according to claim 7, wherein: Displaying the item search indication information on the image captured by the camera includes: Obtaining a display position of an object that matches the selected object position information on an image captured by the camera that matches the depth image; Determining a display position of the augmented reality indication information according to the display position; The item search indication information is displayed on the image captured by the camera according to the display position.

9. The method according to any one of claims 6 to 8, wherein: The preset item database is obtained based on the method according to any one of claims 1-5.

10. A device for storing item location information, comprising: a first acquiring unit configured to acquire target depth data, wherein the target depth data is obtained based on depth data captured for an area where an object to be identified is located; an identification unit configured to identify the object to be identified based on the target depth data; a first generating unit configured to generate object location information of the object to be identified based on a relative relationship between a position of a capturing device of the depth data and an identified position of the object to be identified, wherein the object location information includes altitude information for displaying object search indication information, and the search indication information includes prompt information for indicating that the object to be found is not on a current horizontal plane; The associating unit is configured to associate and store the object location information of the object to be identified with the matching object tag.

11. A device for displaying item search indication information, comprising: a selection unit configured to, in response to receiving the item search information, select, from a preset item database, item location information that matches the item tag indicated by the item search information, wherein the item database includes item location information and corresponding item tags, and the item location information includes altitude information; a second acquiring unit, configured to acquire position information of the camera in response to detecting that the camera is turned on; a second generating unit configured to generate item search indication information based on a distance between a position indicated by the camera position information and a position indicated by the selected item position information, and further configured to: in response to determining that the altitude difference between the altitude indicated by the camera position information and the position indicated by the selected item position information is greater than a preset altitude threshold, generate augmented reality prompt information indicating that the item to be found is not on a current horizontal plane; The display unit is configured to display the item search indication information on the image captured by the camera.

12. An electronic device comprising: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 9.

13. A computer-readable medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images