Target article retrieval method and device, electronic equipment and storage medium

Abstract

The invention provides a target article retrieval method and device, electronic equipment and a storage medium, and relates to the field of artificial intelligence such as large models, deep learning, computer vision and cloud computing. The method comprises the steps that retrieval requirement description information and an initial image are obtained, the retrieval requirement description information is description information about a to-be-retrieved target article, and the initial image is an image obtained after a preset article placement area is shot; segmenting an object in the initial image to obtain segmented images serving as segmentation results; and in response to the identified segmented image matched with the retrieval demand description information, taking the matched segmented image as a target image, and generating a retrieval result corresponding to the target article according to the target image and returning the retrieval result.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to target item retrieval methods, devices, electronic devices and storage media in the fields of large models, deep learning, computer vision and cloud computing. Background Art

[0002] Currently, users have the need to search for target items in many scenarios. For example, when users are shopping in a supermarket, if they want to find the target product they want to buy, they can only search manually by visual observation, which requires a lot of manpower and time costs and is inefficient. Summary of the invention

[0003] The present invention provides a target item retrieval method, device, electronic device and storage medium.

[0004] A target object retrieval method, comprising:

[0005] Acquire retrieval requirement description information and an initial image, wherein the retrieval requirement description information is description information about the target item to be retrieved, and the initial image is an image obtained by photographing a predetermined item placement area;

[0006] Segmenting the objects in the initial image to obtain segmented images as segmentation results;

[0007] In response to identifying a segmented image that matches the retrieval requirement description information, the matching segmented image is used as a target image, and a retrieval result corresponding to the target item is generated and returned based on the target image.

[0008] A target object retrieval method, comprising:

[0009] Acquire the search requirement description information input by the user, and acquire the initial image, wherein the search requirement description information is the description information about the target item to be searched, and the initial image is the image obtained by photographing the predetermined item placement area;

[0010] Sending the search requirement information and the initial image to a cloud service;

[0011] In response to obtaining the retrieval result returned by the cloud service, the retrieval result is displayed to the user, the retrieval result is a retrieval result corresponding to the target object generated by the cloud service based on the target image, the target image is a segmented image identified by the cloud service from the segmentation result and matching the retrieval requirement description information, and the segmentation result includes: each segmented image obtained after the cloud service segments the objects in the initial image.

[0012] A target object retrieval device comprises: a first acquisition module, an image segmentation module and a retrieval processing module;

[0013] The first acquisition module is used to acquire retrieval requirement description information and an initial image, wherein the retrieval requirement description information is description information about a target item to be retrieved, and the initial image is an image obtained by photographing a predetermined item placement area;

[0014] The image segmentation module is used to segment the objects in the initial image to obtain segmented images as segmentation results;

[0015] The retrieval processing module is used to respond to identifying a segmented image that matches the retrieval requirement description information, taking the matching segmented image as a target image, and generating a retrieval result corresponding to the target item based on the target image and returning it.

[0016] A target item retrieval device comprises: a second acquisition module, an information sending module and a result processing module;

[0017] The second acquisition module is used to acquire the search requirement description information input by the user and acquire the initial image, wherein the search requirement description information is the description information about the target item to be searched, and the initial image is an image obtained by photographing a predetermined item placement area;

[0018] The information sending module is used to send the search requirement information and the initial image to the cloud service;

[0019] The result processing module is used to display the search result to the user in response to obtaining the search result returned by the cloud service, wherein the search result is a search result corresponding to the target object generated by the cloud service according to the target image, and the target image is a segmented image identified by the cloud service from the segmentation result and matching the search requirement description information, and the segmentation result includes: each segmented image obtained after the cloud service segments the objects in the initial image.

[0020] An electronic device, comprising:

[0021] at least one processor; and

[0022] a memory communicatively connected to the at least one processor; wherein,

[0023] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the method described above.

[0024] A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the method as described above.

[0025] A computer program product comprises a computer program / instruction, wherein the computer program / instruction implements the method described above when executed by a processor.

[0026] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0028] Figure 1 This is a flow chart of a first embodiment of the target item retrieval method disclosed in the present invention;

[0029] Figure 2 This is a flow chart of a second embodiment of the target item retrieval method disclosed in the present invention;

[0030] Figure 3 Schematic diagram of the structure of the first embodiment 300 of the target item retrieval device disclosed in the present invention;

[0031] Figure 4 Schematic diagram of the structure of the second embodiment 400 of the target item retrieval device disclosed in the present invention;

[0032] Figure 5 A schematic block diagram of an electronic device 500 that can be used to implement an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0033] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0034] In addition, it should be understood that the term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0035] Figure 1Flow chart of the first embodiment of the target item retrieval method disclosed in the present invention. Figure 1 As shown, the following specific implementation methods are included.

[0036] In step 101, retrieval requirement description information and an initial image are obtained, wherein the retrieval requirement description information is description information about a target item to be retrieved, and the initial image is an image obtained by photographing a predetermined item placement area.

[0037] In step 102, the objects in the initial image are segmented to obtain segmented images as segmentation results.

[0038] In step 103, in response to identifying a segmented image that matches the retrieval requirement description information, the matching segmented image is used as a target image, and a retrieval result corresponding to the target item is generated and returned based on the target image.

[0039] By adopting the scheme described in the above method embodiment, the retrieval results (i.e., search results) of the target items to be retrieved (i.e., to be found) can be automatically generated based on the retrieval requirement description information and the initial image through operations such as image segmentation and matching recognition, thereby simplifying manual operations, saving manpower and time costs, and improving processing efficiency.

[0040] In practical applications, Figure 1 The execution subject of the illustrated embodiment may be a cloud service, i.e., a background service, and the cloud service may obtain the search requirement description information and the initial image from the smart terminal. For example, the user may aim the camera of the smart terminal at a supermarket shelf to take an image, and may input the search requirement description information into the smart terminal, i.e., input the description information of the target product (the product in this case) to be searched, such as "I want to find ** brand potato chips", or "I want to find ** flavor potato chips, in bags", etc. Correspondingly, the smart terminal may send the search requirement description information and the initial image to the cloud service.

[0041] For the acquired initial image, the cloud service can segment the items therein and obtain segmented images as the segmentation results. For example, the initial image can be used as the input of a pre-trained arbitrary segmentation model (SAM, SegmentAnything Model) to obtain segmented images output by the segmentation model. Taking a supermarket shelf as an example, each segmented image can include one commodity, such as a bag of melon seeds, or multiple similar commodities, such as a pile of candies. Generally speaking, the objects in the same segmented image belong to the same closed contour.

[0042] Afterwards, a segmented image matching the retrieval requirement description information can be identified from the segmentation results, and the matching segmented image can be used as the target image. In actual applications, it is also possible that a segmented image matching the retrieval requirement description information cannot be identified. In this case, there is no restriction on how to handle it. For example, a prompt message "no relevant results found" can be directly returned to the smart terminal so that the smart terminal can display it to the user.

[0043] In some embodiments of the present disclosure, in response to determining that the retrieval requirement description information is in text form, keywords can be extracted from the retrieval requirement description, and a target image can be recognized based on the keywords. In response to determining that the retrieval requirement description information is in voice form, the retrieval requirement description information can be converted from voice to text, and keywords can be extracted from the conversion result, and then the target image can be recognized based on the keywords.

[0044] In other words, it can support users to input the search requirement description information in text form, and it can also support users to input the search requirement description information in voice form, which is very flexible and convenient, and can meet the different usage needs of users.

[0045] If it is in text form, keywords can be directly extracted from the retrieval requirement description information. If it is in voice form, the retrieval requirement description information can first be converted from voice to text to obtain the required conversion result, and then keywords can be extracted from the conversion result.

[0046] The extracted keywords may include the name and features of the target item. For example, if the search requirement description information is "I want to find ** brand potato chips", the extracted keywords may be "** brand potato chips" (the name of the target item). For another example, if the search requirement description information is "I want to find ** flavor potato chips, packaged", the extracted keywords may include "potato chips", "** flavor" and "packaged". In addition, there is no restriction on how to extract keywords. For example, the extraction may be based on rules, or the extraction may be performed using a pre-trained extraction model.

[0047] After the keywords are acquired, the target image can be identified according to the keywords. In some embodiments of the present disclosure, the following processing can be performed for each segmented image: in response to identifying that the segmented image includes the target object according to the keywords, the segmented image is added to the first image set, and in response to determining that the first image set is not empty, the segmented image in the first image set is determined as the target image.

[0048] For example, assuming that there are 30 segmented images in the segmentation result, for the sake of convenience, they are called segmented images 1 to 30 respectively. For segmented image 1, if the target object is included in segmented image 1 according to the keyword, then segmented image 1 can be added to the first image set. If the target object is not included in segmented image 1 according to the keyword, segmented image 1 will not be added to the first image set. Similarly, for segmented image 2, if the target object is included in segmented image 2 according to the keyword, then segmented image 2 can be added to the first image set. If the target object is not included in segmented image 2 according to the keyword, then segmented image 2 will not be added to the first image set. The processing methods of other segmented images are similar and will not be repeated. Assuming that the first image set is not empty after all segmented images have been processed, the segmented images in the first image set can be determined as target images.

[0049] By adopting the above processing method, all target images can be identified from the segmentation results, thereby improving the comprehensiveness and accuracy of the recognition results.

[0050] Alternatively, in some embodiments of the present disclosure, when target images are identified based on keywords, a statistical parameter may be set with an initial value of 0, and each segmented image may be traversed in sequence according to a predetermined order, and the following processing may be performed for each traversed segmented image: in response to identifying that the segmented image includes a target object based on the keyword, the segmented image is added to the second image set, and the value of the statistical parameter is increased by 1; in response to determining that the latest value of the statistical parameter is equal to N, the traversal is terminated, and the segmented image in the second image set is determined as the target image, where N is a positive integer; in response to determining that the latest value of the statistical parameter is equal to N, the traversal is terminated, and the segmented image in the second image set is determined as the target image, where N is a positive integer; The value is less than N, and it is determined that there is a segmented image that has not been traversed, continue to traverse the next segmented image, in response to determining that the latest value of the statistical parameter is less than N, and it is determined that all the segmented images have been traversed, the segmented image in the second image set is determined as the target image; in response to identifying that the segmented image does not include the target object according to the keyword, and it is determined that there is a segmented image that has not been traversed, continue to traverse the next segmented image, in response to identifying that the segmented image does not include the target object according to the keyword, and it is determined that all the segmented images have been traversed, the segmented image in the second image set is determined as the target image.

[0051] The specific value of N can be determined according to actual needs, and N can represent an upper limit set by the user. In addition, the specific order of the predetermined order can also be determined according to actual needs.

[0052] Assume that there are 30 segmented images in the segmentation result, which are respectively called segmented images 1 to 30 for ease of description, and assume that they are traversed in the order of segmented images 1 to 30. Taking segmented image 15 as an example, if it is identified that the target object is included in segmented image 15 based on the keyword, then segmented image 15 can be added to the second image set, and the value of the statistical parameter can be increased by 1. If it is determined that the value of the updated statistical parameter is equal to N, segmented images 16 to 30 may not be traversed, but the segmented image in the second image set may be directly determined as the target image. If it is determined that the value of the updated statistical parameter is less than N, then segmented image 16 may continue to be traversed, or, if it is identified that the target object is not included in segmented image 15 based on the keyword, segmented image 16 may continue to be traversed. If after all 30 segmented images are traversed, the value of the statistical parameter is less than or equal to N, then the segmented image in the second image set may also be determined as the target image.

[0053] By adopting the above processing method, once it is determined that the number of identified segmented images including the target object reaches N, the traversal can be terminated, and the segmented image in the current second image set can be directly determined as the target image, thereby eliminating the need to identify all the segmented images, thereby improving processing efficiency and avoiding returning too much information, affecting the user's information browsing efficiency, etc.

[0054] For any segmented image, the large model can be used to identify whether the target object is included therein. In some embodiments of the present disclosure, a prompt word of the large model can be first constructed according to the keyword, and then the segmented image and the prompt word can be used as input of the large model. In response to obtaining a first output result of the large model, it can be determined that the segmented image includes the target object, and in response to obtaining a second output result of the large model, it can be determined that the segmented image does not include the target object.

[0055] There is no restriction on how to construct the prompt words. For example, assuming that the keywords include "potato chips", "** flavor" and "bag", the constructed prompt words can be "You are ***, please judge whether the items in the picture include bagged potato chips with ** flavor, ****", etc. Accordingly, the large model can generate a first output result indicating "yes" or a second output result indicating "no" based on the input segmented image and prompt words.

[0056] The large model refers to the Large Language Model (LLM), which is a deep learning model trained with a large amount of text data. It can generate natural language text or understand the meaning of language text, etc. It is an important way to artificial intelligence. Accordingly, using the large model to identify whether the segmented image includes the target object can improve the accuracy of the recognition result. In addition, the type of large model in the scheme described in the present disclosure is not limited, as long as it can complete the corresponding function.

[0057] After the target image is determined, a search result corresponding to the target object can be generated according to the target image, and the search result can be returned to the smart terminal.

[0058] In some embodiments of the present disclosure, the segmentation results obtained by segmenting the initial image may also include: position information of each segmented image, wherein the position information is the position information in the initial image. Accordingly, when the retrieval result corresponding to the target item is generated based on the target image, the position information of the target image may be returned as the retrieval result.

[0059] For example, assuming that the target image only includes the segmented image 15, the location information of the segmented image 15 can be returned to the smart terminal as a retrieval result so that the smart terminal can display it to the user, so that the user can know the location of the target object and it is convenient for the user to pick it up.

[0060] In addition, in actual applications, cloud services can be further divided into access services, voice conversion services, segmentation services, extraction services, and retrieval services. Among them, the access service can obtain the retrieval requirement description information and the initial image from the smart terminal, and can return the retrieval results to the smart terminal. In addition, other services can be called, such as calling the voice conversion service to convert the voice-based retrieval requirement description information into text, calling the extraction service to extract keywords from the text-based retrieval requirement description information, calling the segmentation service to segment the initial image, and calling the retrieval service to identify the target image from the segmentation results based on the keywords and return the retrieval results generated based on the target image. Among them, the access service, voice conversion service, extraction service, and retrieval service can all be implemented using serverless technology.

[0061] The above mainly describes the solution of the present disclosure from the cloud service side, and the following further describes the solution of the present disclosure from the smart terminal side.

[0062] Figure 2 Flow chart of the second embodiment of the target item retrieval method disclosed in the present invention. Figure 2 As shown, the following specific implementation methods are included.

[0063] In step 201, the search requirement description information input by the user is obtained, and an initial image is obtained. The search requirement description information is the description information about the target item to be searched, and the initial image is an image obtained by shooting a predetermined item placement area.

[0064] In step 202, the search requirement information and the initial image are sent to the cloud service.

[0065] In step 203, in response to obtaining the retrieval results returned by the cloud service, the retrieval results are displayed to the user. The retrieval results are retrieval results corresponding to the target objects generated by the cloud service based on the target image. The target image is a segmented image that matches the retrieval requirement description information identified by the cloud service from the segmentation results. The segmentation results include: each segmented image obtained after the cloud service segments the objects in the initial image.

[0066] By adopting the scheme described in the above method embodiment, the search results of the target items to be searched can be automatically generated based on the search requirement description information and the initial image through operations such as image segmentation and matching recognition, thereby simplifying manual operations, saving manpower and time costs, and improving processing efficiency.

[0067] In practical applications, Figure 2 The execution subject of the illustrated embodiment may be a smart terminal. The smart terminal may obtain the search requirement description information input by the user, and may obtain the initial image, and send the search requirement description information and the initial image to the cloud service.

[0068] The smart terminal can also obtain the search results generated and returned by the cloud service. The cloud service can segment the objects in the initial image to obtain segmented images as segmentation results, and in response to identifying the segmented images that match the search requirement description information, the matching segmented images can be used as target images, and the search results corresponding to the target objects can be generated based on the target images and returned to the smart terminal.

[0069] In some embodiments of the present disclosure, the segmentation result may further include: position information of each segmented image, where the position information is position information in the initial image. Accordingly, the retrieval result may include: position information of the target image.

[0070] Furthermore, the intelligent terminal can display the search results to the user. In some embodiments of the present disclosure, the initial image can be annotated to obtain an optimized image, and the annotation can include: annotating the target image with a rectangular frame according to the position information of the target image, and then the optimized image can be displayed to the user.

[0071] For example, assuming that the target image only includes the segmented image 15, the segmented image 15 can be marked with a rectangular frame in the initial image based on the position information of the segmented image 15 to obtain an optimized image, and the optimized image can be displayed to the user, so that the user can intuitively view the location of the target object, making it easier for the user to pick it up.

[0072] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present disclosure is not limited by the described order of actions, because according to the present disclosure, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present disclosure. In addition, for parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0073] The above is an introduction to the method embodiment. The following is a further explanation of the scheme disclosed in the present invention through an apparatus embodiment.

[0074] Figure 3 FIG. 3 is a schematic diagram of the composition structure of the first embodiment 300 of the target item retrieval device described in the present disclosure. Figure 3 As shown, it includes: a first acquisition module 301, an image segmentation module 302 and a retrieval processing module 303.

[0075] The first acquisition module 301 is used to acquire retrieval requirement description information and an initial image. The retrieval requirement description information is description information about a target item to be retrieved, and the initial image is an image obtained by photographing a predetermined item placement area.

[0076] The image segmentation module 302 is used to segment the objects in the initial image to obtain segmented images as segmentation results.

[0077] The retrieval processing module 303 is used to respond to identifying the segmented image matching the retrieval requirement description information, taking the matching segmented image as the target image, and generating a retrieval result corresponding to the target item based on the target image and returning it.

[0078] In some embodiments of the present disclosure, the retrieval processing module 303, in response to determining that the retrieval requirement description information is in text form, can perform keyword extraction on the retrieval requirement description and can identify the target image based on the keywords. In response to determining that the retrieval requirement description information is in voice form, the retrieval requirement description information can be converted from voice to text, and keywords can be extracted from the conversion result, and the target image can be identified based on the keywords.

[0079] Among them, when identifying the target image according to keywords, in some embodiments of the present disclosure, the retrieval processing module 303 may perform the following processing for each segmented image: in response to identifying that the segmented image includes the target object according to the keywords, the segmented image is added to the first image set; in response to determining that the first image set is not empty, the segmented image in the first image set is determined as the target image.

[0080] Alternatively, in some embodiments of the present disclosure, when the retrieval processing module 303 performs target image recognition according to keywords, a statistical parameter may be set, with an initial value of 0, and each segmented image may be sequentially traversed in a predetermined order, and the following processing may be performed for each traversed segmented image: in response to recognizing that the segmented image includes a target object according to the keyword, the segmented image is added to the second image set, and the value of the statistical parameter is increased by 1; in response to determining that the latest value of the statistical parameter is equal to N, the traversal is terminated, and the segmented image in the second image set is determined as the target image, where N is a positive integer; in response to determining that the latest value of the statistical parameter is equal to N, the traversal is terminated, and the segmented image in the second image set is determined as the target image, where N is a positive integer; In response to determining that the latest value of the statistical parameter is less than N and that all the segmented images have been traversed, the segmented image in the second image set is determined as the target image; in response to identifying that the segmented image does not include the target object according to the keyword and determining that there is a segmented image that has not been traversed, the next segmented image is continued to be traversed; in response to identifying that the segmented image does not include the target object according to the keyword and determining that all the segmented images have been traversed, the segmented image in the second image set is determined as the target image.

[0081] For any segmented image, the large model can be used to identify whether the target object is included therein. In some embodiments of the present disclosure, the retrieval processing module 303 can first construct a prompt word of the large model based on the keyword, and then use the segmented image and the prompt word as input of the large model. In response to obtaining the first output result of the large model, it can be determined that the segmented image includes the target object, and in response to obtaining the second output result of the large model, it can be determined that the segmented image does not include the target object.

[0082] In addition, in some embodiments of the present disclosure, the segmentation results obtained by segmenting the initial image may also include: position information of each segmented image, where the position information is the position information in the initial image. Accordingly, when the retrieval processing module 303 generates a retrieval result corresponding to the target item based on the target image, the position information of the target image may be returned as the retrieval result.

[0083] Figure 4 FIG. 4 is a schematic diagram of the structure of the second embodiment 400 of the target item retrieval device described in the present disclosure. Figure 4As shown, it includes: a second acquisition module 401, an information sending module 402 and a result processing module 403.

[0084] The second acquisition module 401 is used to obtain the search requirement description information input by the user and to obtain the initial image. The search requirement description information is the description information about the target item to be searched, and the initial image is the image obtained by shooting a predetermined item placement area.

[0085] The information sending module 402 is used to send the search requirement information and the initial image to the cloud service.

[0086] The result processing module 403 is used to display the search results to the user in response to obtaining the search results returned by the cloud service. The search results are search results corresponding to the target items generated by the cloud service based on the target image. The target image is a segmented image that matches the search requirement description information identified by the cloud service from the segmentation results. The segmentation results include: each segmented image obtained after the cloud service segments the objects in the initial image.

[0087] In some embodiments of the present disclosure, the segmentation result may further include: position information of each segmented image, where the position information is position information in the initial image. Accordingly, the retrieval result may include: position information of the target image.

[0088] Further, the result processing module 403 may display the search results to the user. In some embodiments of the present disclosure, the result processing module 403 may annotate the initial image to obtain an optimized image, and the annotation may include: annotating the target image with a rectangular frame according to the position information of the target image, and then displaying the optimized image to the user.

[0089] The specific working processes of the above-mentioned device embodiments can refer to the relevant descriptions in the above-mentioned method embodiments and will not be repeated here.

[0090] The solution disclosed in the present disclosure can be applied to the field of artificial intelligence, especially to the fields of large models, deep learning, computer vision and cloud computing. Artificial intelligence is a discipline that studies how computers can simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It has both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing. Artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, as well as machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.

[0091] In addition, the search requirement description information and images in the embodiments of the present disclosure are not for a specific user and do not reflect the personal information of a specific user. In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good customs.

[0092] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0093] Figure 5 A schematic block diagram of an electronic device 500 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0094] like Figure 5 As shown, the device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 to a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0095] A number of components in the device 500 are connected to the I / O interface 505, including: an input unit 506, such as a keyboard, a mouse, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a disk, an optical disk, etc.; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 allows the device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0096] The computing unit 501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI, Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSP, Digital Signal Processing), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 501 performs the various methods and processes described above, such as the methods described in the present disclosure. For example, in some embodiments, the methods described in the present disclosure may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, one or more steps of the methods described in the present disclosure may be executed. Alternatively, in other embodiments, the computing unit 501 may be configured to execute the method described in the present disclosure in any other appropriate manner (for example, by means of firmware).

[0097] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard parts (ASSPs), system on chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0098] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0099] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory (EPROM, Electronically Programmable Read-Only Memory), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM, Compact Disc Read-Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0100] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a cathode ray tube (CRT) or a liquid crystal display (LCD) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0101] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by digital data communication (e.g., a communication network) in any form or medium. Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0102] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0103] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0104] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A target item retrieval method, comprising: Acquire retrieval requirement description information and an initial image, wherein the retrieval requirement description information is description information about the target item to be retrieved, and the initial image is an image obtained by photographing a predetermined item placement area; Segmenting the objects in the initial image to obtain segmented images as segmentation results; In response to identifying a segmented image that matches the retrieval requirement description information, the matching segmented image is used as a target image, and a retrieval result corresponding to the target item is generated and returned based on the target image.

2. The method according to claim 1, wherein: In response to identifying a segmented image matching the retrieval requirement description information, taking the matching segmented image as a target image comprises: In response to determining that the retrieval requirement description information is in text form, extracting keywords from the retrieval requirement description, and identifying the target image according to the keywords; In response to determining that the retrieval requirement description information is in voice form, the retrieval requirement description information is converted from voice to text, keywords are extracted from the conversion result, and the target image is recognized based on the keywords.

3. The method according to claim 2, wherein: The identifying of the target image according to the keyword includes: For each segmented image, the following processing is performed respectively: in response to identifying that the segmented image includes the target object according to the keyword, adding the segmented image to the first image set; In response to determining that the first image set is not empty, a segmented image in the first image set is determined as the target image.

4. The method according to claim 2, wherein: The identifying of the target image according to the keyword includes: Set the statistical parameters, with the initial value of 0, and traverse each segmented image in the predetermined order. For each segmented image traversed, perform the following processing: In response to identifying that the target object is included in the segmented image according to the keyword, adding the segmented image to the second image set and adding 1 to the value of the statistical parameter; in response to determining that the latest value of the statistical parameter is equal to N, ending the traversal and determining the segmented image in the second image set as the target image, where N is a positive integer; in response to determining that the latest value of the statistical parameter is less than N and that there is a segmented image that has not been traversed, continuing to traverse the next segmented image; in response to determining that the latest value of the statistical parameter is less than N and that all segmented images have been traversed, determining the segmented image in the second image set as the target image; In response to identifying that the target object is not included in the segmented image according to the keyword, and determining that there is a segmented image that has not been traversed, continuing to traverse the next segmented image; in response to identifying that the target object is not included in the segmented image according to the keyword, and determining that all segmented images have been traversed, determining the segmented image in the second image set as the target image.

5. The method according to claim 3 or 4, wherein: The step of identifying that the target object is included in the segmented image according to the keyword comprises: constructing a prompt word of a large model according to the keyword, taking the segmented image and the prompt word as inputs of the large model, and determining that the target object is included in the segmented image in response to obtaining a first output result of the large model; The step of identifying that the target object is not included in the segmented image based on the keyword comprises: constructing the prompt word based on the keyword, using the segmented image and the prompt word as input of the large model, and determining that the target object is not included in the segmented image in response to obtaining a second output result of the large model.

6. The method according to any one of claims 1 to 4, wherein: The segmentation result also includes: position information of each segmented image, where the position information is position information in the initial image; The returning of the retrieval result corresponding to the target item generated according to the target image includes: returning the location information of the target image as the retrieval result.

7. A target object retrieval method, comprising: Acquire the search requirement description information input by the user, and acquire the initial image, wherein the search requirement description information is the description information about the target item to be searched, and the initial image is the image obtained by photographing the predetermined item placement area; Sending the search requirement information and the initial image to a cloud service; In response to obtaining the retrieval result returned by the cloud service, the retrieval result is displayed to the user, the retrieval result is a retrieval result corresponding to the target object generated by the cloud service based on the target image, the target image is a segmented image identified by the cloud service from the segmentation result and matching the retrieval requirement description information, and the segmentation result includes: each segmented image obtained after the cloud service segments the objects in the initial image.

8. The method according to claim 7, wherein: The segmentation result also includes: position information of each segmented image, where the position information is position information in the initial image; The retrieval result includes: location information of the target image.

9. The method according to claim 8, wherein: The presenting the search result to the user comprises: Annotating the initial image to obtain an optimized image, wherein the annotation includes: annotating the target image with a rectangular frame according to the position information of the target image; The optimized image is presented to the user.

10. A target object retrieval device, comprising: A first acquisition module, an image segmentation module and a retrieval processing module; The first acquisition module is used to acquire retrieval requirement description information and an initial image, wherein the retrieval requirement description information is description information about a target item to be retrieved, and the initial image is an image obtained by photographing a predetermined item placement area; The image segmentation module is used to segment the objects in the initial image to obtain segmented images as segmentation results; The retrieval processing module is used to respond to identifying a segmented image that matches the retrieval requirement description information, taking the matching segmented image as a target image, and generating a retrieval result corresponding to the target item based on the target image and returning it.

11. The device according to claim 10, wherein: In response to determining that the retrieval requirement description information is in text form, the retrieval processing module extracts keywords from the retrieval requirement description and identifies the target image based on the keywords. In response to determining that the retrieval requirement description information is in voice form, the retrieval requirement description information is converted from voice to text, and keywords are extracted from the conversion result, and the target image is identified based on the keywords.

12. The device according to claim 11, wherein The retrieval processing module performs the following processing on each segmented image: in response to identifying that the segmented image includes the target object based on the keyword, the segmented image is added to the first image set; in response to determining that the first image set is not empty, the segmented image in the first image set is determined as the target image.

13. The device according to claim 11, wherein: The retrieval processing module sets a statistical parameter with an initial value of 0, and traverses each segmented image in turn according to a predetermined order, and performs the following processing for each traversed segmented image: in response to identifying that the segmented image includes the target object according to the keyword, the segmented image is added to the second image set, and the value of the statistical parameter is increased by 1; in response to determining that the latest value of the statistical parameter is equal to N, the traversal is terminated, and the segmented image in the second image set is determined as the target image, N is a positive integer; in response to determining that the latest value of the statistical parameter is less than N and that there is a segmented image that has not been traversed, the traversal continues to the next segmented image; in response to determining that the latest value of the statistical parameter is less than N and that all segmented images have been traversed, the segmented image in the second image set is determined as the target image; In response to identifying that the target object is not included in the segmented image according to the keyword and determining that there are segmented images that have not been traversed, continue to traverse the next segmented image; in response to identifying that the target object is not included in the segmented image according to the keyword and determining that all segmented images have been traversed, determine the segmented image in the second image set as the target image.

14. The device according to claim 12 or 13, wherein: The retrieval processing module constructs a prompt word of the large model based on the keywords, takes the segmented image and the prompt word as input of the large model, determines that the segmented image includes the target object in response to obtaining a first output result of the large model, and determines that the segmented image does not include the target object in response to obtaining a second output result of the large model.

15. The device according to any one of claims 10 to 13, wherein: The segmentation result also includes: position information of each segmented image, where the position information is position information in the initial image; The retrieval processing module returns the location information of the target image as the retrieval result.

16. A target object retrieval device, comprising: A second acquisition module, an information sending module and a result processing module; The second acquisition module is used to acquire the search requirement description information input by the user and acquire the initial image, wherein the search requirement description information is the description information about the target item to be searched, and the initial image is an image obtained by photographing a predetermined item placement area; The information sending module is used to send the search requirement information and the initial image to the cloud service; The result processing module is used to display the search result to the user in response to obtaining the search result returned by the cloud service, wherein the search result is a search result corresponding to the target object generated by the cloud service according to the target image, and the target image is a segmented image identified by the cloud service from the segmentation result and matching the search requirement description information, and the segmentation result includes: each segmented image obtained after the cloud service segments the objects in the initial image.

17. The device according to claim 16, wherein: The segmentation result also includes: position information of each segmented image, where the position information is position information in the initial image; The retrieval result includes: location information of the target image.

18. The device according to claim 17, wherein: The result processing module annotates the initial image to obtain an optimized image, wherein the annotation includes: annotating the target image with a rectangular frame according to the position information of the target image, and displaying the optimized image to the user.

19. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 9.

20. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to make a computer execute the method according to any one of claims 1 to 9.

21. A computer program product, comprising a computer program / instruction, wherein when the computer program / instruction is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images