Method, device and storage medium for feeding back search results

Abstract

The present application provides a method, device and storage medium for feeding back search results, and relates to the field of terminal technology. It includes: obtaining a first image including M objects, and for N objects among the M objects, when N is greater than or equal to 2, determining the arrangement order of the N objects. The arrangement order of any object is determined based on any one or more of the scene intention weight, the confidence score, and the object relationship score. The scene intention weight is used to indicate the probability of any object being searched in the scene corresponding to the first image. The confidence score is the similarity between any object and the image in the image library. The object relationship score is used to indicate the importance of any object in the first image. According to the arrangement order of the N objects, the search results of some or all of the N objects are fed back. The search results of objects with good search results and that the user may be interested in are fed back to the user first, so that the feedback has a certain degree of pertinence.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to a method, device and storage medium for feeding back search results. Background Art

[0002] Visual search uses images as search input sources to obtain various search results such as related images and texts of objects in the images. More and more users use visual search technology on electronic devices to meet their search needs for some objects.

[0003] Currently, in the visual search process, the electronic device acquires an image, and then performs image detection processing on the image to determine one or more objects in the image. The electronic device searches for the determined objects to obtain search results. After obtaining the search results, the electronic device feeds back the search results of each object to the user in the order of the position of each object in the image.

[0004] However, the search results fed back to the user are search results for each object in the first image, and the feedback is less targeted, resulting in a lower accuracy of the search results fed back to the user. Summary of the invention

[0005] The present application provides a method, device and storage medium for feeding back search results, which solve the problem of low accuracy of search results fed back to users in the prior art.

[0006] In order to achieve the above objectives, this application adopts the following technical solutions:

[0007] In a first aspect, a method for feeding back search results is provided, the method comprising:

[0008] Acquire a first image, where the first image includes M objects, where M is an integer greater than or equal to 2;

[0009] For N objects among the M objects, when N is greater than or equal to 2, determining an arrangement order of the N objects, where N is a positive integer less than or equal to M;

[0010] The arrangement order of any one of the N objects is determined based on any one or more of a scene intention weight, a confidence score, and an object relationship score, wherein the scene intention weight is used to indicate the probability of any one of the objects being searched in the scene corresponding to the first image, the confidence score is the similarity between any one of the objects and the images in the image library, and the object relationship score is used to indicate the importance of any one of the objects in the first image;

[0011] According to the arrangement order of the N objects, search results of some or all of the N objects are fed back.

[0012] The present application screens and sorts N objects according to their arrangement order, so as to give priority to feeding back to the user the search results of objects with good search results and that the user may be interested in, so that the feedback has a certain degree of pertinence, thereby achieving the effect of improving the accuracy of the search results fed back in the end.

[0013] As an example of the present application, any one or more of the scene intention weight, confidence score, and object relationship score of any one of the N objects is determined by:

[0014] Performing image detection processing on the first image by using a target detection model to obtain an object region of a first object in the first image and / or an object category of the first object, wherein the first object is any one of the N objects;

[0015] A confidence score of the first object is determined based on an object region of the first object in the first image and an object category of the first object, and / or an object relationship score of the first object is determined based on the object region of the first object in the first image, and / or a scene intention weight of the first object is determined based on the object category of the first object.

[0016] The present application performs image detection processing on a first image through a target detection model to obtain an object area and / or an object category of a first object in the first image, thereby determining at least one of a scene intention weight, a confidence score, and an object relationship score of the first object based on the image area and / or the object category of the first object, and then uses at least one of the scene intention weight, confidence score, and object relationship score of each object to screen and sort the objects in the first image, so as to accurately provide feedback to the user on objects that the user may be interested in.

[0017] As an example of the present application, determining a confidence score of the first object based on an object region of the first object in the first image and an object category of the first object includes:

[0018] determining a library of images corresponding to an object category of the first object;

[0019] determining a similarity between an object region of the first object in the first image and each of a plurality of images included in the image library;

[0020] The maximum similarity among the determined multiple similarities is used as the confidence score of the first object.

[0021] In this way, the search object is screened out from the first image according to the confidence score, so that the intention understanding corresponds to the search result, which can improve the accuracy of the intention search.

[0022] As an example of the present application, the first image includes a plurality of regions, and each of the plurality of regions has a preset score for indicating the importance of the location of each region, and the preset scores of at least two of the plurality of regions are different;

[0023] The determining the object relationship score of the first object based on the object region of the first object in the first image comprises:

[0024] determining a position importance value of the first object in the first image based on a preset score of each area included in the object area of ​​the first object in the first image;

[0025] An object relationship score of the first object is determined based on the position importance value of the first object.

[0026] By determining the position importance values ​​of N objects in the first image in the two-dimensional space, determining the object relationship score based on the position importance value, and filtering the search objects from the first image according to the object relationship score, so as to only perform the intended search on the more important objects in the first image, thereby providing the user with the search results of the search objects of interest, and improving the user experience.

[0027] As an example of the present application, before determining the object relationship score of the first object based on the position importance value of the first object, the method further includes:

[0028] When the first image includes a reference object, based on an object region of the first object in the first image and an object region of the reference object in the first image, obtaining a distance between the first object and the reference object in the first image as a dependency degree value of the first object;

[0029] The determining the object relationship score of the first object based on the position importance value of the first object includes:

[0030] An object relationship score of the first object is determined based on the position importance value and the dependency relationship value of the first object.

[0031] By determining the position importance values ​​of N objects in the first image in the two-dimensional space, and determining the degree of affiliation between each object and the reference object, and then determining the object relationship score of each object based on the position importance value and the degree of affiliation value, the search intention of the object is associated with the object attribute, thereby giving priority to recommending search results of search objects that the user may be interested in, thereby improving the user experience.

[0032] As an example of the present application, determining the scene intention weight of the first object based on the object category of the first object includes:

[0033] determining a scene category of the first image;

[0034] The scene intent weight of the first object is determined based on the scene category of the first image and the object category of the first object, as well as the correspondence between the scene category, the object category and the scene intent weight.

[0035] By determining the scene category of the first image, and then determining the search intent ranking of the search object in the first image according to the scene category, the search results of the search object in the first image are displayed according to the search intent ranking. In this way, the search results are combined with the scene, so that the search results are closer to the scene, and the search results are prevented from being separated from the scene of the first image, so that the search results displayed to the user are more consistent with the scene of the first image, so as to give priority to recommending search results that the user may be interested in, thereby improving the user experience.

[0036] As an example of the present application, the quality scores of the N objects in the first image are greater than or equal to a quality score threshold, and the quality score of any one of the objects is determined based on the blurriness and / or completeness of the any one of the objects.

[0037] By filtering out some objects with poor image quality, the intent understanding is associated with the object attributes, and the search processing of some objects that do not match the search intent is avoided, the effectiveness and accuracy of the search can be improved, thereby improving the user experience.

[0038] As an example of the present application, feeding back search results of some or all of the N objects according to the arrangement order of the N objects includes:

[0039] Obtain object tags included in search results of some or all of the N objects;

[0040] Based on the acquired object tags, query information corresponding to some or all of the N objects is generated, where the query information is used to prompt whether information associated with the object tags needs to be acquired;

[0041] According to the arrangement order of the N objects, query information of some or all of the N objects is displayed to feed back search results of some or all of the N objects.

[0042] Generate query information based on object tags, and use the query information to feed back search results to users, so that users can quickly understand the feedback results and improve the simplicity of the feedback interface.

[0043] As an example of the present application, the N objects include a second object and a third object, the arrangement order of the second object is before the arrangement order of the third object, and the display order of the search results corresponding to the second object on the display screen is before the display order of the search results corresponding to the third object on the display screen.

[0044] The search results of the objects in the front row of the arrangement order are displayed in front, that is, the search results of the objects in the front row of the arrangement order are displayed first, so that the user can view the search results displayed in the front row of the display screen first, and the search results of the objects that the user may be interested in are fed back to the user first, thereby achieving the effect of improving the user experience.

[0045] In a second aspect, a device for feeding back search results is provided, the device comprising:

[0046] An acquisition module, configured to acquire a first image, wherein the first image includes M objects, where M is an integer greater than or equal to 2;

[0047] a determination module, configured to determine, for N objects among the M objects, an arrangement order of the N objects when N is greater than or equal to 2, where N is a positive integer less than or equal to M;

[0048] The arrangement order of any one of the N objects is determined based on any one or more of a scene intention weight, a confidence score, and an object relationship score, wherein the scene intention weight is used to indicate the probability of any one of the objects being searched in the scene corresponding to the first image, the confidence score is the similarity between any one of the objects and the images in the image library, and the object relationship score is used to indicate the importance of any one of the objects in the first image;

[0049] A feedback module is used to feed back search results of some or all of the N objects according to the arrangement order of the N objects.

[0050] As an example of the present application, the determining module is used to:

[0051] Performing image detection processing on the first image by using a target detection model to obtain an object region of a first object in the first image and / or an object category of the first object, wherein the first object is any one of the N objects;

[0052] A confidence score of the first object is determined based on an object region of the first object in the first image and an object category of the first object, and / or an object relationship score of the first object is determined based on the object region of the first object in the first image, and / or a scene intention weight of the first object is determined based on the object category of the first object.

[0053] As an example of the present application, the determining module is used to:

[0054] determining a library of images corresponding to an object category of the first object;

[0055] determining a similarity between an object region of the first object in the first image and each of a plurality of images included in the image library;

[0056] The maximum similarity among the determined multiple similarities is used as the confidence score of the first object.

[0057] As an example of the present application, the first image includes a plurality of regions, and each of the plurality of regions has a preset score for indicating the importance of the location of each region, and the preset scores of at least two of the plurality of regions are different; the determining module is used to:

[0058] determining a position importance value of the first object in the first image based on a preset score of each area included in the object area of ​​the first object in the first image;

[0059] An object relationship score of the first object is determined based on the position importance value of the first object.

[0060] As an example of the present application, the determining module is used to:

[0061] When the first image includes a reference object, based on an object region of the first object in the first image and an object region of the reference object in the first image, obtaining a distance between the first object and the reference object in the first image as a dependency degree value of the first object;

[0062] An object relationship score of the first object is determined based on the position importance value and the dependency relationship value of the first object.

[0063] As an example of the present application, the determining module is used to:

[0064] determining a scene category of the first image;

[0065] The scene intent weight of the first object is determined based on the scene category of the first image and the object category of the first object, as well as the correspondence between the scene category, the object category and the scene intent weight.

[0066] As an example of the present application, the quality scores of the N objects in the first image are greater than or equal to a quality score threshold, and the quality score of any one of the objects is determined based on the blurriness and / or completeness of the any one of the objects.

[0067] As an example of the present application, the feedback module is used to:

[0068] Obtain object tags included in search results of some or all of the N objects;

[0069] Based on the acquired object tags, query information corresponding to some or all of the N objects is generated, where the query information is used to prompt whether information associated with the object tags needs to be acquired;

[0070] According to the arrangement order of the N objects, query information of some or all of the N objects is displayed to feed back search results of some or all of the N objects.

[0071] As an example of the present application, the N objects include a second object and a third object, the arrangement order of the second object is before the arrangement order of the third object, and the display order of the search results corresponding to the second object on the display screen is before the display order of the search results corresponding to the third object on the display screen.

[0072] According to a third aspect, an electronic device is provided, wherein the structure of the electronic device includes a processor and a memory, wherein the memory is used to store a program supporting the electronic device to execute the method for feeding back search results described in any one of the first aspects, and to store data involved in implementing the method for feeding back search results described in any one of the first aspects; the processor is configured to execute the program stored in the memory; the electronic device may also include a communication bus, wherein the communication bus is used to establish a connection between the processor and the memory.

[0073] In a fourth aspect, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium is run on a computer, the computer executes the method as described in any one of the above-mentioned first aspects.

[0074] In a fifth aspect, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute the method for feeding back search results described in the first aspect.

[0075] The technical effects obtained by the above-mentioned second, third, fourth and fifth aspects are similar to the technical effects obtained by the corresponding technical means in the above-mentioned first aspect, and will not be repeated here.

[0076] The technical solution provided by this application can at least bring the following beneficial effects:

[0077] A first image is obtained, the first image includes M objects, and M is an integer greater than or equal to 2. For N objects among the M objects, when the number of the N objects is multiple, the arrangement order of the N objects is determined, and according to the arrangement order of the N objects, the search results of some or all of the N objects are fed back. Among them, the arrangement order of any object among the N objects is determined based on any one or more of the scene intention weight, the confidence score, and the object relationship score. The scene intention weight is used to indicate the probability of any object being searched in the scene corresponding to the first image, the confidence score is the similarity between any object and the image in the image library, and the object relationship score is used to indicate the importance of any object in the first image. In this way, the N objects are screened and sorted according to the arrangement order of the N objects, so that the search results of objects with good search results and that the user may be interested in are fed back to the user first, so that the feedback has a certain degree of pertinence, thereby improving the accuracy of the search results fed back in the end. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0079] Figure 2 A software structure block diagram of an electronic device provided in an embodiment of the present application;

[0080] Figure 3 This is a diagram showing an interface of a browser running in a mobile phone provided in an embodiment of the present application;

[0081] Figure 4 A schematic diagram of an intent search process provided in an embodiment of the present application;

[0082] Figure 5 A schematic diagram of a feedback effect of a search result provided in an embodiment of the present application;

[0083] Figure 6 A schematic diagram of another feedback effect of search results provided in an embodiment of the present application;

[0084] Figure 7A system architecture diagram for feeding back search results provided in an embodiment of the present application;

[0085] Figure 8 A schematic diagram of an implementation framework for feedback search results provided in an embodiment of the present application;

[0086] Fig. 9 A flowchart of a method for feeding back search results provided in an embodiment of the present application;

[0087] Fig.10 A schematic diagram of an image detection process provided in an embodiment of the present application;

[0088] Fig.11 A schematic diagram of a regional image of an object obtained after image processing provided in an embodiment of the present application;

[0089] Fig.12 A schematic diagram of the fuzziness and completeness of an object provided in an embodiment of the present application;

[0090] Fig.13 A schematic diagram of a confidence score of an object provided in an embodiment of the present application;

[0091] Fig.14 A flowchart of another method for feeding back search results provided in an embodiment of the present application;

[0092] Fig.15 A schematic diagram of an image segmentation effect provided in an embodiment of the present application;

[0093] Fig.16 A schematic diagram of a Gaussian distribution of a preset score provided in an embodiment of the present application;

[0094] Fig.17 A flowchart of another method for feeding back search results provided in an embodiment of the present application;

[0095] Fig.18 A flowchart of another method for feeding back search results provided in an embodiment of the present application;

[0096] Fig.19 A schematic diagram of the structure of a device for feeding back search results provided in an embodiment of the present application. DETAILED DESCRIPTION

[0097] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0098] It should be understood that the "multiple" mentioned in this application refers to two or more. In the description of this application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in order to facilitate the clear description of the technical solution of this application, the words "first" and "second" are used to distinguish between the same items or similar items with basically the same functions and effects. Those skilled in the art can understand that the words "first" and "second" do not limit the quantity and execution order, and the words "first" and "second" do not limit them to be different.

[0099] The method provided in the embodiment of the present application can be applied to electronic devices. In one embodiment, the electronic device can install and run an application (application, APP) with a visual search function, for example, the APP can be a shopping APP, a celebrity search APP, a browser, etc. As an example, the electronic device can be a device such as a wearable device, a terminal device, etc. Exemplarily, wearable devices can include but are not limited to smart watches, smart bracelets, smart brooches, smart eye masks, and smart glasses. Terminal devices can include but are not limited to mobile phones, tablet computers, augmented reality (AR) / virtual reality (VR) devices, ultra-mobile personal computers (UMPC), laptops, netbooks, and personal digital assistants (PDA).

[0100] See also Figure 1 , Figure 1 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application.

[0101] The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0102] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0103] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0104] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0105] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use instructions or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0106] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0107] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple groups of I2C buses. The processor 110 may be coupled to the touch sensor 180K, the charger, the flash, the camera 193, etc. through different I2C bus interfaces. For example: the processor 110 may be coupled to the touch sensor 180K through the I2C interface, so that the processor 110 communicates with the touch sensor 180K through the I2C bus interface, thereby realizing the touch function of the electronic device 100.

[0108] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to achieve communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 via the I2S interface to achieve the function of answering a call through a Bluetooth headset.

[0109] The PCM interface can also be used for audio communication, sampling, quantizing and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface to realize the function of answering calls via a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0110] The UART interface is a universal serial data bus for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is generally used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to implement the Bluetooth function. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 through the UART interface to implement the function of playing music through a Bluetooth headset.

[0111] The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to implement the shooting function of the electronic device 100. The processor 110 and the display screen 194 communicate via the DSI interface to implement the display function of the electronic device 100.

[0112] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 with the camera 193, the display 194, the wireless communication module 160, the audio module 170, the sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0113] The USB interface 130 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used to transfer data between the electronic device 100 and a peripheral device. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other electronic devices, such as AR devices, etc.

[0114] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present invention is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0115] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the electronic device 100. While the charging management module 140 is charging the battery 142, it may also power the electronic device through the power management module 141.

[0116] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0117] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0118] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0119] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0120] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0121] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0122] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0123] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0124] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0125] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0126] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0127] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0128] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0129] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0130] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0131] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0132] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0133] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0134] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0135] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0136] The receiver 170B, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or voice message, the voice can be received by placing the receiver 170B close to the human ear.

[0137] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to microphone 170C to input the sound signal into microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the sound source, realize directional recording function, etc.

[0138] The earphone interface 170D is used to connect a wired earphone and can be a USB interface 130 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0139] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 194, the electronic device 100 detects the touch operation intensity according to the pressure sensor 180A. The electronic device 100 can also calculate the touch position according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, an instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, an instruction to create a new short message is executed.

[0140] The gyro sensor 180B can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyro sensor 180B. The gyro sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyro sensor 180B detects the angle of the electronic device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 100 through reverse movement to achieve anti-shake. The gyro sensor 180B can also be used for navigation and somatosensory game scenes.

[0141] The air pressure sensor 180C is used to measure air pressure. In some embodiments, the electronic device 100 calculates the altitude through the air pressure value measured by the air pressure sensor 180C to assist positioning and navigation.

[0142] The magnetic sensor 180D includes a Hall sensor. The electronic device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip leather case. In some embodiments, when the electronic device 100 is a flip phone, the electronic device 100 can detect the opening and closing of the flip cover according to the magnetic sensor 180D. Then, according to the detected opening and closing state of the leather case or the opening and closing state of the flip cover, the flip cover can be automatically unlocked.

[0143] The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device 100 in all directions (generally three axes). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the electronic device 100 and applied to applications such as horizontal and vertical screen switching and pedometers.

[0144] The distance sensor 180F is used to measure the distance. The electronic device 100 can measure the distance by infrared or laser. In some embodiments, when shooting a scene, the electronic device 100 can use the distance sensor 180F to measure the distance to achieve fast focusing.

[0145] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The electronic device 100 emits infrared light outward through the light emitting diode. The electronic device 100 uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device 100. When insufficient reflected light is detected, the electronic device 100 can determine that there is no object near the electronic device 100. The electronic device 100 can use the proximity light sensor 180G to detect that the user holds the electronic device 100 close to the ear to talk, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used in leather case mode and pocket mode to automatically unlock and lock the screen.

[0146] The ambient light sensor 180L is used to sense the ambient light brightness. The electronic device 100 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the electronic device 100 is in a pocket to prevent accidental touches.

[0147] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.

[0148] The temperature sensor 180J is used to detect temperature. In some embodiments, the electronic device 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold, the electronic device 100 reduces the performance of a processor located near the temperature sensor 180J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the electronic device 100 heats the battery 142 to avoid abnormal shutdown of the electronic device 100 due to low temperature. In other embodiments, when the temperature is lower than another threshold, the electronic device 100 performs a boost on the output voltage of the battery 142 to avoid abnormal shutdown caused by low temperature.

[0149] The touch sensor 180K is also called a "touch panel". The touch sensor 180K can be set on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the electronic device 100, which is different from the position of the display screen 194.

[0150] The bone conduction sensor 180M can obtain a vibration signal. In some embodiments, the bone conduction sensor 180M can obtain a vibration signal of a vibrating bone block of the vocal part of the human body. The bone conduction sensor 180M can also contact the human pulse to receive a blood pressure beat signal. In some embodiments, the bone conduction sensor 180M can also be set in an earphone and combined into a bone conduction earphone. The audio module 170 can parse out a voice signal based on the vibration signal of the vibrating bone block of the vocal part obtained by the bone conduction sensor 180M to realize a voice function. The application processor can parse the heart rate information based on the blood pressure beat signal obtained by the bone conduction sensor 180M to realize a heart rate detection function.

[0151] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The electronic device 100 may receive key input and generate key signal input related to user settings and function control of the electronic device 100.

[0152] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0153] Indicator 192 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0154] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to and separated from the electronic device 100 by inserting it into the SIM card interface 195 or pulling it out from the SIM card interface 195. The electronic device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the electronic device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0155] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. In the embodiment of the present invention, the Android system of the layered architecture is taken as an example to exemplify the software structure of the electronic device 100.

[0156] Figure 2 It is a software structure block diagram of the electronic device 100 according to an embodiment of the present application.

[0157] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system library, and the kernel layer.

[0158] The application layer can include a series of application packages.

[0159] like Figure 2 As shown, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc. In addition, the application package may also include an application with a visual search function.

[0160] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0161] like Figure 2As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0162] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0163] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0164] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0165] The phone manager is used to provide communication functions of the electronic device 100, such as management of call status (including connecting, hanging up, etc.).

[0166] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0167] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0168] Android Runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0169] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0170] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0171] The system library may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0172] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0173] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0174] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0175] A 2D graphics engine is a drawing engine for 2D drawings.

[0176] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0177] The following is an illustrative description of the workflow of the software and hardware of the electronic device 100 in conjunction with capturing a photo scene.

[0178] When the touch sensor 180K receives a touch operation, the corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into raw input events (including touch coordinates, timestamp of the touch operation, and other information). The raw input events are stored in the kernel layer. The application framework layer obtains the raw input events from the kernel layer and identifies the controls corresponding to the input events. For example, if the touch operation is a touch single-click operation and the control corresponding to the single-click operation is the control of the camera application icon, the camera application calls the interface of the application framework layer to start the camera application, and then starts the camera driver by calling the kernel layer to capture static images or videos through the camera 193.

[0179] Based on the above Figure 1 and Figure 2 The electronic device provided in the illustrated embodiment will now be briefly introduced into the application scenarios involved in the embodiments of the present application. Here, a mobile phone is taken as an example:

[0180] The mobile phone obtains a poster. As an example, the poster can be taken by the user triggering the mobile phone, or it can be an image in the mobile phone's gallery. For example, assuming that a browser with a visual search function is installed in the mobile phone, when the user wants to perform a visual search based on the poster, he can click the browser in the mobile phone. In response to the user's click operation on the browser, the mobile phone starts the browser. As an example, Figure 3 As shown in FIG. 1 (a), the browser provides an add option 31, and the user can trigger the add option 31. Figure 3 As shown in FIG. 1( b), in response to the user's triggering operation on adding option 31, the mobile phone displays the "shooting" option and the "album" option. In one embodiment, when the user wants to shoot a poster with the mobile phone, the user can trigger Figure 3 In (b), the mobile phone detects the user's triggering operation on the "shooting" option and opens the camera, so that the poster can be captured by the camera. In another embodiment, when the user wants to use a poster in the gallery, he can click the "album" option in (b). The mobile phone detects the user's triggering operation on the "album" option and opens the gallery. The user selects a poster from the gallery, so that the mobile phone can obtain the poster based on the user's selection.

[0181] As an example, after the mobile phone obtains the poster, the poster can be displayed in the display interface. For example, the display effect of the poster is as follows: Figure 4 As shown in (a) in the figure. In one embodiment, a "search" option 32 may also be provided in the display interface. The user may click on the "search" option 32. After the mobile phone detects that the user has triggered the "search" option 32, it performs a visual search based on the poster to obtain search results for the search object in the poster. The search object is an object that the user may be interested in in the poster. The search object is determined by the mobile phone. The specific determination process can be found in the embodiment described below. In one embodiment, the features of the search object in the poster include at least one of the following (1)-(3): (1) clear outline, (2) high completeness, (3) position close to the center area of ​​the poster, that is, the position is relatively central. The number of search objects in the poster may be one or more, for example Figure 4 The search object in the poster shown in Figure (a) may include face B, or may also include face B and mobile phone A.

[0182] It should be noted that here, only the "search" option 32 is provided in the display interface for the user to trigger the mobile phone to perform visual search as an example. In another embodiment, the user can also trigger the mobile phone to perform visual search in other ways. For example, after the mobile phone displays the poster, the user can also "shake" the mobile phone. Accordingly, after the mobile phone detects the user's "shake" operation, it performs a visual search based on the obtained poster.

[0183] After obtaining the search results, the mobile phone feeds back the search results to the user. As an example of the present application, since the search results of the search object may contain a lot of content, in order to improve the simplicity of the display effect and enable the user to quickly and intuitively understand the search results, the mobile phone can feed back to the user through the form of an inquiry scene. For example, the inquiry scene may include inquiry information related to the search object in the poster. For example, assuming that the poster is as follows: Figure 4 As shown in Figure (a) in FIG. 1 , and the mobile phone determines that the search objects in the poster include mobile phone A and face B, the inquiry scene may include inquiry information related to mobile phone A and face B, such as Figure 4 As shown in Figure (b) of FIG. 1 , the inquiry information fed back by the mobile phone to the user includes “Do you want to know the relevant information of the celebrity A Xing?” and “Do you want to buy this mobile phone A?” In one embodiment, when there are multiple search objects, the mobile phone displays the inquiry information of the multiple search objects in descending order of priority, such as Figure 4 As shown in Figure (b), since the priority of face B is higher than that of mobile phone A, the query information of face B is displayed before the query information of mobile phone A.

[0184] As an example, the user can click on any of the displayed query information to make the mobile phone display other content in the search results related to the query information clicked by the user. For example, when the user wants to know more about the celebrity A Xing in the poster, he can click on the query information "Do you want to know more about the celebrity A Xing?" After the mobile phone detects the triggering operation of the query information "Do you want to know more about the celebrity A Xing?", it can obtain and display other content in the search results of the celebrity A Xing, such as the introduction information of the celebrity A Xing, the works published, etc. The displayed results are as follows: Figure 4 As shown in Figure (c) in the figure. For another example, when a user wants to buy mobile phone A in the poster, he can click on the query information "Do you want to buy this mobile phone A?" In one embodiment, when the mobile phone detects the triggering operation of the query information "Do you want to buy this mobile phone A?", it can obtain and display other content in the search results of mobile phone A, such as displaying multiple images of mobile phone A. The display results are as follows: Figure 4 As shown in Figure (d) in FIG. In another embodiment, after the mobile phone detects a click operation on the inquiry information "Do you want to buy this mobile phone A?", if the purchase address information of mobile phone A can be obtained, for example, the purchase address information of mobile phone A is included in the search results, the purchase address information can also be displayed to the user. When the user's trigger operation on the purchase address information is detected, the page jumps to the purchase page, so that the user can purchase mobile phone A on the purchase page, avoiding the need to manually enter the search information of mobile phone A to search again when the user wants to purchase mobile phone A, thereby improving the user experience.

[0185] It should be noted that in the above embodiment, only the entire inquiry information is triggerable as an example. In another embodiment, part of the text in the inquiry information may be triggerable, for example, "Do you want to know the relevant information of celebrity A Xing?" The "Celebrity A Xing" in this inquiry information is triggerable. When the user wants to know more relevant information of celebrity A Xing, he can click on the "Celebrity A Xing" text to trigger the mobile phone to display other content in the search results of celebrity A Xing. For another example, "Do you want to buy this mobile phone A?" The "mobile phone A" and / or "buy" in this inquiry information are triggerable. When the user wants to know about mobile phone A in the poster, he can click on "mobile phone A" or "buy" to trigger the mobile phone to display other content in the search results of mobile phone A in the poster.

[0186] As another example of the present application, the feedback form of the search result can also be as follows: Figure 5 As shown, that is, the triggerable items can be identified in the area where each search object in one or more search objects in the poster is located, and the triggerable items corresponding to each search object can be identified in other areas other than the poster in the display interface. As an example, when there are multiple search objects, the triggerable items of the multiple search objects can be identified in order of priority from high to low. When the user wants to know more relevant information related to the search object, the user can click on the triggerable item corresponding to the search object. After the mobile phone detects the user's triggering operation on the triggerable item, it can display other content in the search results of the search object. For example, the display effect is as follows: Figure 4 As shown in Figure (c) or (d).

[0187] As an example, the triggerable items corresponding to different search objects can be marked with different colors and / or different shapes. For example, the triggerable items corresponding to face B can be marked with blue dots, and the triggerable items corresponding to mobile phone A can be marked with red dots. For another example, the triggerable items corresponding to face B can be marked with dots, and the triggerable items corresponding to mobile phone A can be marked with squares.

[0188] It is worth mentioning that by providing users with feedback on search results through triggerable items, the simplicity of the interface display can be improved.

[0189] In one embodiment, in the display interface, in addition to the poster, the area near the triggerable items marked by other area marks can also display corresponding query information. For example, the query information "Know about star Xing" can be displayed behind the triggerable item corresponding to the celebrity A Xing, and the query information "Buy this phone A" can be displayed behind the triggerable item corresponding to the mobile phone A in the poster. In this way, the user can intuitively perceive the search results corresponding to the triggerable items, thereby improving the user experience.

[0190] It should be noted that the above Figure 5 This is just an example of identifying triggerable items in the area where each search object is located and other areas in the poster. In another embodiment, triggerable items can also be identified only in the area where each search object is located in the poster, or triggerable items can also be identified only in the other areas, which is not limited in the embodiment of the present application.

[0191] As another example of the present application, the mobile phone can also display the search results of the search object in different rows. When there are multiple search objects, the mobile phone can list and display the search results of different search objects in different rows in the order of the priorities of the multiple search objects from high to low, that is, the search results of one or more search objects in the poster are displayed in different rows. For example, please refer to Figure 6 , the mobile phone displays the search results of face B in the first row and the search results of mobile phone A in the second row. In one embodiment, since the content that can be displayed in each row is limited, one row may not be able to display all the search results of the search object. In this case, the mobile phone can also identify the triggerable items of the search object. For example, the corresponding triggerable items are identified in the area where the face of face B is located, and the corresponding triggerable items are identified in the area where mobile phone A is located in the poster. For another example, the triggerable items are displayed in the row corresponding to the search object. In this way, when the user wants to know more about other content in the search results related to the search object, he can click on the triggerable item corresponding to the search object to make the mobile phone display other content.

[0192] It should be noted that the above-mentioned several feedback forms of search results are only exemplary. In another embodiment, the mobile phone may also use other methods to feedback the search results of the search object to the user, which is not limited in this embodiment of the present application.

[0193] After introducing the application scenarios involved in the embodiments of the present application, the system architecture involved in the embodiments of the present application will be introduced next. Figure 7 , Figure 7This is a schematic diagram of a system architecture according to an exemplary embodiment. As an example of the present application, the system architecture includes a terminal side and a cloud side. In one embodiment, the execution subject of the terminal side is an electronic device, the execution subject of the cloud side is a server, and a communication connection is established between the electronic device and the server.

[0194] As an example, Figure 7 As shown, the electronic device includes an intention recognition module, a data association module, and a content layout and processing module. In implementation, the electronic device obtains a first image. As an example, the first image can be obtained by taking a picture with a camera, or it can also be an image uploaded from an album. The intention recognition module is used to determine the search object in the first image through operations such as image detection and processing, and the server performs an intention search on the search object in the first image, and the server returns the search results. In addition, when the first image includes multiple search objects, the intention recognition module is also used to determine the priority of the multiple search objects. The data association module is used to obtain and display the search results returned by the server. In one embodiment, when the number of search objects is multiple, the data association module is also used to obtain the priority of the multiple search objects determined by the intention recognition module, and sort the obtained search results based on the priority of the multiple search objects, which can also be understood as sorting the multiple search objects. The content layout and processing module is used to layout and feedback the sorted multiple search results.

[0195] Optionally, the electronic device may further include an image preprocessing module, which is used to preprocess the image captured or uploaded by the electronic device. Exemplarily, the image preprocessing module may perform resizing, denoising and other preprocessing to obtain a first image. In this way, preprocessing the captured or uploaded image can make the obtained first image more consistent with the processing requirements of the intent recognition module. For example, resizing can solve the problem of inconsistent image sizes caused by different camera specifications. For example, denoising can improve the accuracy of image detection processing by the intent recognition module, thereby improving the accuracy of visual search.

[0196] The server is used to determine and return search results. The server is provided with databases corresponding to different vertical domains, one vertical domain corresponds to one object type. Exemplarily, different vertical domains may include but are not limited to celebrity vertical domains, commodity vertical domains, and scenic spot vertical domains. In one embodiment, the database corresponding to each vertical domain includes but is not limited to image library, semantic information, and work information. The image library includes at least one image of the search object, for example, at least one image may be an image taken from different angles of the search object. Semantic information is the language description information of the search object, for example, it may include introduction information of the search object. Work information includes information about works such as articles published by people and videos released. At least one image, semantic information, and work information of each search object included in the database are interrelated. Exemplarily, the search results returned by the server to the electronic device generally include at least one image of the search object in the image library, and semantic information and work information associated with at least one image.

[0197] In one possible implementation, the search results of a search object may include multiple data. In this case, before the server feeds back the search results to the electronic device, it may also reorder the multiple data in the search results of the search object to determine the display order of the multiple data in the search results of the search object, and feed back the reordered search results to the electronic device. As an example, the server may sort the multiple data in the search results of the search object according to a preset strategy. Among them, the preset strategy can be set in advance according to actual needs. For example, if the search results of the search object include multiple images, the preset strategy may be to sort by shooting angle, such as front images before images at other angles.

[0198] In a possible implementation of the present application, the intent recognition module may also perform a primary search during the process of performing intent search through the server, that is, perform a preliminary operation. During the initial search, the server determines the similarity between the object in the first image and the image in the image library to determine whether there is a search result matching the object in the image library, thereby screening the object in the first image through the initial search, and then determining the search object in the first image according to the screening result. The specific implementation can be found in the embodiment shown below. In one embodiment, the server determines the similarity based on the image preliminary library in the database. Among them, the image preliminary library is a sub-library of the corresponding image library. As an example, a part of the images can be extracted from the image library corresponding to each vertical domain in advance according to a preset strategy to obtain the image preliminary library corresponding to each vertical domain. For example, for the image library corresponding to any vertical domain, an image of each object is extracted from the image library to obtain the image preliminary library corresponding to any vertical domain. In this way, when determining the similarity, it is possible to avoid the need to match the object with all the images in the image library, which can save the server's computing power.

[0199] It is worth mentioning that the embodiment of the present application is implemented through the interaction between the electronic device and the server, that is, the electronic device obtains search results and performs an initial search through the server. This avoids the need to set up databases corresponding to different vertical domains in the electronic device, thereby reducing the operating burden of the electronic device and further improving the operating speed of the electronic device.

[0200] Of course, it should be noted that the above is an example of the method provided in the embodiment of the present application being implemented by the interaction between the end side and the cloud side. In another embodiment, the method can also be implemented by the end side alone, that is, it can be implemented by the electronic device alone. For example, a database corresponding to different vertical domains is provided in the electronic device. In this way, when determining the search results and the initial search, the electronic device can execute based on its own database, avoiding the need to rely on the server to achieve visual search. In another embodiment, the method can also be implemented by the server alone.

[0201] After introducing the application scenarios and system architecture involved in the embodiments of the present application, the specific implementation of the method for determining search intent provided in the embodiments of the present application is introduced. The method can be applied to the above Figure 7 The execution subject of the embodiment of the present application can be an electronic device or a server. If the execution subject is a server, the display process involved (for example Fig. 9 The display process involved in step 905 in the illustrated embodiment may be omitted, or the display process may also be performed by other devices equipped with a display device.

[0202] The following takes the execution subject as an electronic device as an example. As an example of the present application, the intention recognition module in the electronic device includes three branches, each branch corresponds to a processing unit, that is, the intention recognition module includes three processing units, namely the first processing unit, the second processing unit, and the third processing unit. Figure 8 , the first processing unit is used to perform the operation of the branch corresponding to the left virtual box, the second processing unit is used to perform the operation of the branch corresponding to the middle virtual box, and the third processing unit is used to perform the operation in the branch corresponding to the right virtual box. In one embodiment, the related process of determining the arrangement order of objects involved in the method can be independently implemented by any one of the three processing units. In another embodiment, the related process of determining the arrangement order of objects involved in the method can be implemented in combination by any two of the three processing units. In yet another embodiment, the related process of determining the arrangement order of objects involved in the method can also be implemented in combination by the three processing units. For ease of understanding, the implementation methods of the three processing units are introduced below through the following embodiments.

[0203] Please refer to Fig. 9 , Fig. 9 This is a flow chart of a method for feeding back a search result according to an exemplary embodiment. Here, the relevant process of determining the arrangement order of objects involved in the method is implemented separately by the second processing unit as an example. Specifically, it may include part or all of the following contents:

[0204] Step 901: Acquire a first image.

[0205] As an example, the first image is obtained by taking a picture by a camera of an electronic device, or the first image is an image from a gallery. As another example, the first image can also be obtained by preprocessing a taken or uploaded image. The specific implementation of obtaining the first image can be found in the above text and will not be repeated here.

[0206] The first image includes M objects, where M is an integer greater than or equal to 2.

[0207] Step 902: Determine object regions of N objects and / or object categories of N objects included in the first image.

[0208] Wherein, N is a positive integer less than or equal to M.

[0209] The object region of each object refers to the object region of each object in the first image.

[0210] Please refer to Figure 8 , the electronic device performs image detection processing on the first image through the intention recognition module to determine the object area of ​​each object and / or the object category of each object in the N objects included in the first image. As an example, the intention recognition module includes a target detection model, so that the first image is subjected to image detection processing through the target detection model to obtain the object area of ​​each object and / or the object category of each object. The target detection model can be a pre-trained detection model that can be used to detect the object area and / or object category of an object in any image in any image.

[0211] For example, assuming that the first image is Fig.10 As shown in (a) of FIG. 1 , after the first image is input into the target detection model, the target detection model determines the position information and object category of each object among the N objects included in the first image, and the position information is used to indicate the object area of ​​the object in the first image. As an example, the electronic device uses an object frame to mark each object in the first image according to the position information determined by the target detection model, thereby obtaining a second image. For example, the second image is as follows Fig.10 As shown in Figure (b), after image detection processing, it is determined that the N objects included in the first image are human faces, mobile phones and clothes, and the object categories of the N objects are human faces, commodities and clothes respectively.

[0212] In one embodiment, the target detection model can be obtained by pre-training the model to be trained based on the object detection sample set. As an example, each object detection sample in the object detection sample set is an object image marked with an object category label. For example, the object detection sample set may include face image samples, clothing image samples, commodity image samples, landmark image samples, animal image samples, plant image samples, etc.

[0213] As an example of the present application, the model to be trained may be a model combining mobilenetV3 and a single shot multibox detector (SSD), that is, a model using mobilenetV3 as a network basis and a structure of SSD. The network structure of mobilenetV3 can be described by Table 1.

[0214] Table 1

[0215]

[0216] As an example of the present application, after the electronic device determines the object regions of N objects and / or the object categories of N objects included in the first image, the object region of each of the N objects can be cut out from the first image. Fig.10 After the object region of each object in the first image shown in FIG. (b) is cut, the object region of each object is obtained as shown in FIG. Fig.11 As shown, Fig.11 The object shown in (a) is a "mobile phone". Fig.11 The object shown in (b) is a "human face". Fig.11 The object shown in (c) is "clothing".

[0217] In a possible implementation manner, the N objects included in the first image are the M objects in the first image, that is, N equals M. In another possible implementation manner, the N objects included in the first image are determined after quality screening, that is, before determining the object areas of the N objects and / or the object categories of the N objects, the M objects in the first image may also be quality screened to determine the N objects.

[0218] As an example, the quality scores of N objects in the first image are greater than or equal to the quality score threshold, and the quality score of any one of the N objects is determined based on the blurriness and / or integrity of any one of the objects. The quality score threshold can be set according to actual needs. In one embodiment, the quality screening process can include the following sub-steps (1)-(2):

[0219] (1) Determine a quality score for each of the M objects included in the first image.

[0220] In one embodiment, due to reasons such as shooting angle, the image quality of one or some objects in the first image may be poor. For example, the first image may include passers-by or other objects, which are relatively blurred in the first image, or only a small part appears. For example, the first image may only include part of the body of a passer-by. In most cases, such objects are not the objects of concern to the user. Therefore, the electronic device can determine the quality score of each of the M objects included in the first image through the second processing unit, so as to filter some of the M objects included in the first image, thereby filtering out objects with poor image quality.

[0221] As an example, the image quality of any object can be measured by the blurriness and / or completeness of any object, that is, the second processing unit determines the blurriness and / or completeness of each object in the M objects included in the first image, thereby determining the quality score of each object. The blurriness can be described by the clarity of the outline of the object, that is, if the outline of an object is clear, it can be determined that the blurriness of the object is low, and conversely, if the outline of an object is unclear, it can be determined that the blurriness of the object is high. Completeness refers to the proportion of the unobstructed part of the object to the whole.

[0222] In one embodiment, the second processing unit may use the Laplace variance algorithm to determine the blurriness of the object. For example, the Laplace mask may be used to perform a convolution operation on the pixel values ​​in the region where the object is located in the first image, and then the variance may be calculated and the variance may be score-mapped to obtain the blurriness of the object. Exemplarily, this may be implemented by the following code:

[0223] frame = cv2.imread(img)

[0224] resImg=cv2.resize(frame,(800,900),interpolation=cv2.INTER_CUBIC)

[0225] img2gray=cv2.cvtColor(resImg,cv2.COLOR_BGR2GRAY)

[0226] res=cv2.Laplacian(img2gray,cv2.CV_64F)

[0227] score = res.var()

[0228] In the above code, the object area of ​​any object among the M objects is read, and then the size of the area where any object is located is adjusted to a fixed size, and the pixel values ​​in the area where any object is located after the size adjustment are grayed out, that is, converted into a single-channel grayscale image, and the Laplace standard deviation is calculated based on the single-channel grayscale image to obtain the blurriness of any object.

[0229] In one embodiment, the integrity of any one of the M objects can be determined by a pre-trained integrity prediction model. Exemplarily, for any one of the M objects included in the first image, the object region of any one of the M objects can be input into the integrity prediction model, and the integrity prediction model performs prediction processing and outputs the integrity of any one of the objects.

[0230] In one embodiment, the completeness prediction model can be obtained by training the model to be trained based on the completeness training sample set, the completeness training sample set can include multiple completeness training samples, each completeness training sample includes an object sample image and the completeness of the object sample image, and the model to be trained can be an untrained mobilenetV3 model. That is, the electronic device can pre-acquire a large number of object samples with different completeness, and then input them into the untrained mobilenetV3 model for model training to obtain the completeness prediction model.

[0231] It should be noted that the completeness prediction model can be a separate model or a model combined with the above-mentioned target detection model, and the embodiments of the present application are not limited to this.

[0232] It should also be noted that the above is only an example of determining the integrity of an object through a completeness prediction model. In another embodiment, the second processing unit may also determine the integrity of the object in other ways. For example, the area of ​​the occluded region of the object may be determined, and the area of ​​the region where the object frame of the object is located may be determined, and then the integrity of the object may be determined based on the two determined areas, which is not limited in the present embodiment of the application.

[0233] Exemplarily, the second processing unit determines the blurriness and integrity of each object in the M objects included in the first image as follows: Fig.12 As shown in the figure, the blurriness and completeness of the mobile phone are as follows Fig.12 As shown in Figure (a), they are 0.022 and 0.86 respectively. The blurriness and completeness of the face are shown in Fig.12 As shown in Figure (b), they are 0.042 and 0.92 respectively. The blurriness and integrity of the clothes are as follows Fig.12 As shown in Figure (c), they are 0.037 and 0.34 respectively.

[0234] After the second processing unit determines the blurriness and / or completeness of each object in the first image, it determines a quality score for each object based on the blurriness and / or completeness of each object in the first image. The quality score is used to indicate the image quality of the object. The quality score is positively correlated with the image quality, that is, the larger the quality score, the better the image quality of the object.

[0235] As an example, taking the determination of the quality score of any object based on the fuzziness and integrity of any object as an example, the second processing unit may set the fuzziness weight and the integrity weight, and then determine the quality score of any object based on the fuzziness and integrity of any object by the following formula (1):

[0236] S i =aF i +bI i (1)

[0237] Among them, S i represents the quality score of object i, a represents the fuzziness weight, a is a negative value, F i represents the fuzziness of object i, b represents the completeness weight, I i Indicates the integrity of object i. The fuzziness weight and the integrity weight can be set according to actual needs, and the embodiment of the present application does not limit this.

[0238] (2) Objects selected from the M objects included in the first image based on the quality score of each object in the M objects are taken as N objects.

[0239] The second processing unit performs filtering processing according to the quality score. For example, for any object among the M objects included in the first image, when the quality score of the any object is lower than a quality score threshold, the any object is filtered out.

[0240] In another embodiment, the M objects included in the first image may be sorted in descending order of quality scores, and then the first N objects ranked high are obtained as N objects, where N is an integer greater than 1.

[0241] It should be noted that the above quality screening process is only exemplary. In another embodiment, the specific implementation of quality screening may include any of the following methods: for any object among the M objects included in the first image, if the blurriness of any object is higher than the blurriness threshold, the any object is filtered out, that is, the objects with relatively blurry images are filtered out; or, if the integrity of any object is lower than the integrity threshold, the any object is filtered out, that is, the objects with lower integrity are filtered out; or, if the blurriness of any object is higher than the blurriness threshold and the integrity of any object is lower than the integrity threshold, the any object is filtered out, that is, the objects that are both blurry and incomplete are filtered out. In addition, the objects remaining after the filtering process are determined as the N objects included in the first image.

[0242] Among them, the fuzziness threshold can be set by the user according to actual needs, or can also be set by default by the electronic device, which is not limited in the embodiments of the present application.

[0243] Among them, the completeness threshold can be set by the user according to actual needs, or can also be set by default by the electronic device, and the embodiments of the present application are not limited to this.

[0244] It is worth mentioning that the embodiment of the present application filters out some objects with poor image quality, so that the intent understanding is associated with the object attributes, avoiding search processing for some objects that do not match the search intent, which can improve the effectiveness and accuracy of the search, thereby improving the user experience.

[0245] It should be noted that if the quality scores of the M objects in the first image are all lower than the quality score threshold, the electronic device ends the intent search. That is, if the M objects included in the first image are all blurred or have low integrity, the electronic device does not perform the intent search, for example, it can prompt the user that the first image is unavailable.

[0246] Step 903: Perform an initial search on the N objects included in the first image.

[0247] As an example, the second processing unit sends a preliminary recruitment request to the server, and the preliminary recruitment request carries the object areas of N objects and the object categories of N objects. After receiving the preliminary recruitment request, the server parses the preliminary recruitment request to obtain the object areas of N objects and the object categories of N objects. This embodiment is illustrated by taking the first object as an example, and the first object is any one of the N objects: the server determines the corresponding vertical domain according to the object category of the first object, and then determines the similarity between the object area of ​​the first object and the image in the image preliminary recruitment library of the determined vertical domain. In implementation, the server extracts object features in the object area of ​​the first object, and obtains object features of each image in the image preliminary recruitment library corresponding to the determined vertical domain, determines the similarity between the extracted object features and the object features of each acquired image, and obtains multiple similarities. The server determines the maximum similarity among the multiple similarities as the confidence score of the first object. As mentioned above, since the image initial search library is a sub-library of the image library, when the similarity is large, it means that there are images matching the first object in the image library, or in other words, there are search results matching the first object in the server's database. Therefore, the confidence score is the similarity between the first object and the image in the image library, which can be used to indicate the degree of match between the search intent of the first object and the search results. As an example and not a limitation, the server determines the label of the image corresponding to the maximum similarity and uses the label as the object label of the first object. The server sends the confidence score and object label of the first object as the result of the initial search (or the initial search result) to the electronic device. Among them, the object label can be used for subsequent selective display by the electronic device.

[0248] It should be noted that, here, the object region and object category of the object are sent to the server, and the server extracts the object features of the object as an example. In another embodiment, the second processing unit can also extract the object features of the object region, and send the object features and object category to the server, so that the server can avoid the need to perform feature extraction on the object region.

[0249] As another example, the electronic device itself is provided with a database. In this case, the second processing unit can determine an image library corresponding to the object category of the first object, determine the similarity between the object area of ​​the first object in the first image and each of the multiple images included in the image library, and then use the maximum similarity among the multiple determined similarities as the confidence score of the first object.

[0250] Step 904: Determine the arrangement order of the N objects according to the result of the initial search.

[0251] The electronic device determines the arrangement order of the N objects according to the confidence score. The arrangement order of any object in the N objects corresponds to the priority, and the higher the arrangement order, the higher the priority.

[0252] Step 905: Feedback search results of some or all of the N objects according to the arrangement order of the N objects.

[0253] That is, the electronic device may feed back search results of all the N objects according to the arrangement order of the N objects, or may select some objects from the N objects and feed back search results of the some objects according to the arrangement order of the N objects.

[0254] For ease of description and understanding, some or all of the N objects to be fed back with search results are referred to as search objects. It is not difficult to understand that the number of search objects may be one or more. That is, the search object includes some of the N objects. In another embodiment, the search object includes all of the N objects.

[0255] As an example, the electronic device selects a preset number of objects with the highest ranking from N objects as search objects. The preset number can be set according to actual needs, for example, the preset number is 2.

[0256] As another example, the electronic device may also determine the search object based on the confidence scores of N objects. For example, for the first object among the N objects, the electronic device compares the confidence score of the first object with the confidence score threshold. If the confidence score of the first object is lower than the confidence score threshold, it means that there may not be a search result matching the first object in the server. Therefore, in order to improve the user experience, that is, to provide users with good search results, the first object may be filtered out, that is, the first object may not be searched for subsequently. On the contrary, if the confidence score of the first object is higher than the confidence score threshold, it means that there is a search result matching the first object in the server. At this time, the first object can be determined as the search object.

[0257] In this way, the search object is screened out from the first image according to the confidence score, so that the intention understanding corresponds to the search result, which can improve the accuracy of the intention search.

[0258] As an example, after the electronic device determines the search object, it sends a search request to the server, and the search request carries the object area and object category of at least one search object. After receiving the search request, the server parses the search request to obtain the object area and object category of at least one search object. Take a search object as an example to illustrate: the server determines the corresponding vertical domain based on the object category of the search object, and obtains the search result of the search object from the database of the determined vertical domain based on the object area. In one embodiment, the database includes an image library and associated information, and the server matches the object area of ​​the search object with the image in the image library to determine at least one image and associated information of the search object. The server uses at least one image and associated information of the search object as the search result of the search object. Afterwards, the server returns the search result of each search object in at least one search object to the electronic device.

[0259] It should be noted that the above is explained by taking the example of sending the object area and object category of at least one search object to the server when the electronic device performs an intention search. In another embodiment, after the server parses the initial request to obtain the object area and object category of N objects, it can generate index information for each of the N objects locally, and establish a mapping relationship between each index information and the object area and object category of each object. The server sends the index information of each of the N objects to the electronic device. In this case, when the electronic device performs an intention search for the search object, it does not need to send the object area and object category of the search object to the server again, but sends the index information of the search object to the server, and the server can directly obtain the object area and object category of the search object from the local based on the index information. In this way, the need for the electronic device to upload the object area and object category again is avoided, which can save traffic.

[0260] When the number of the search object is one, the electronic device directly feeds back the search result of the search object to the user.

[0261] When there are multiple search objects, the search results of the multiple search objects are fed back according to the arrangement order of the multiple search objects, that is, the search results of the search objects with higher priority are displayed first.

[0262] As an example, the specific implementation of feeding back the search results of multiple search objects according to the arrangement order of the multiple search objects may include: obtaining the object label included in the search results of each search object in the multiple search objects, generating query information corresponding to each search object based on the object label of each search object, the query information is used to prompt whether it is necessary to obtain information associated with the object label, and displaying the query information of the multiple search objects in the arrangement order of the multiple search objects to feed back the search results of the multiple search objects.

[0263] The information associated with the object tag of each search object refers to other information included in the search results of each search object except the object tag, for example, semantic information, work information, images, etc. of the search object.

[0264] As an example, the N objects include a second object and a third object, the second object is arranged before the third object, and the search results corresponding to the second object are displayed before the search results corresponding to the third object on the display screen.

[0265] For example, assuming that the search results obtained include the search results for the face B in the first image and the search results for the mobile phone A, please refer to Fig.13 ,like Fig.13 As shown in Figure (a), the confidence score of mobile phone A is 0.886, and Fig.13 As shown in Figure (b), the confidence score of face B is 0.998. The confidence score of face B is higher than that of mobile phone A, that is, the ranking of face B is before that of mobile phone A. In the process of displaying search results, the search results of face B are given priority. For example, the search results of face B can be displayed before the search results of mobile phone A. The display results are as follows: Figure 4 As shown in Figure (b).

[0266] In the embodiment of the present application, the N objects in the first image are initially searched to determine the confidence scores of the N objects in the first image, and then the objects are screened according to the confidence scores of the N objects in the first image to screen out the objects whose intent understanding does not match the search results, thereby improving the accuracy of the intent search. In addition, the screened objects are sorted and displayed based on the confidence scores, so that the search results of the objects that the user may be interested in are displayed first, thereby improving the user experience.

[0267] Please refer to Fig.14 , Fig.14 This is a flow chart of a method for feeding back a search result according to an exemplary embodiment. Here, the method is described by taking the relevant process of determining the arrangement order of objects involved in the method as being implemented by the third processing unit alone as an example. Specifically, it may include part or all of the following contents:

[0268] The specific implementation of step 1401 to step 1402 can refer to the above Fig. 9 Step 901 to step 902 in the embodiment.

[0269] Step 1403: Determine an object relationship score for each of the N objects in the first image.

[0270] The object relationship score of any one of the N objects is used to indicate the importance of any one of the objects in the first image.

[0271] As an example of this application, please refer to Fig.15 , the first image can be divided into multiple areas. Exemplarily, the first image is divided into N*N areas, where N is an integer greater than 1 and can be set according to actual needs. Afterwards, a preset score is set for each area starting from the central area. The preset score of each area is used to indicate the location importance of each area, and the preset scores of at least two areas in the multiple areas are different. Generally, the closer the object to the central area is, the more likely the user is to be interested in it. Therefore, when setting the preset score, the preset score of the area at the center point is set to the highest, and it gradually decreases as it spreads to the surrounding areas. Exemplarily, the preset score of each area can be set according to a Gaussian distribution. The Gaussian distribution of the preset score is as follows: Fig.16 shown.

[0272] The electronic device determines the position importance value of each object among N objects. Taking the first object as an example, the first object is any one of the N objects. The average value of the preset scores of the area included in the object frame of the first object is determined, and the obtained score is used as the position importance value of the first object. Exemplarily, assuming that the N objects in the first image include a face, a mobile phone, and clothes, it is determined by calculation that the position importance value of the face in the first image is 0.78, the position importance value of the mobile phone is 0.7, and the position importance value of the clothes is 0.68.

[0273] Then, the object relationship score of each object is determined based on the position importance value of each object. Exemplarily, the position importance value of each object is determined as the object relationship score of each object, for example, the position importance value of the first object is determined as the object relationship score of the first object.

[0274] In one embodiment of the present application, the first image may also include a reference object, and the reference object can be set according to actual needs, for example, the reference object can be a human body. Taking the reference object as an example, when the first image includes a human body, objects that are usually closer to the human body are objects that the user may be interested in, while objects that are farther away from the human body are objects that the user is not interested in. Therefore, when determining the object relationship scores of N objects in the first image, if the first image includes a reference object, the distance between the first object and the reference object in the first image can also be obtained based on the object area of ​​the first object in the first image and the object area of ​​the reference object in the first image as the degree of affiliation value of the first object. Exemplarily, if the reference object is a human body, when it is determined that the distance between the mobile phone and the human body is 0.42, the degree of affiliation value of the mobile phone is determined to be 0.42.

[0275] As an example, when determining the distance between the first object and the reference object, the distance between the center point of the first object and the center point of the reference object may be determined. In some embodiments, since the distance between the center point of the first object and the center point of the reference object changes when the size of the first image is adjusted, in order to ensure that the degree of attachment values ​​determined for first images of different sizes are the same, the distance may be normalized, for example, the diagonal distance of the first image may be used for normalization.

[0276] Of course, for the reference object, its attachment degree value can be set according to actual needs, for example, it can be set to 1. For example, when the reference object is a human body, the attachment degree value of a face is 1.

[0277] In one embodiment, when the first image includes multiple reference objects, the distance between the first object and each reference object in the first image is determined to obtain multiple distances, and then the minimum distance among the multiple distances is used as the affiliation degree value.

[0278] In one embodiment, when the first image includes a reference object (such as a human body), the electronic device determines the object relationship score of the first object based on the position importance value and the dependency value of the first object. Exemplarily, certain weights can be set for the position importance value and the dependency value, respectively, and then fused in an equal-weighted manner to obtain the object relationship score of the first object. Among them, the weight of the position importance value and the weight of the dependency value can be set according to actual needs, and the embodiment of the present application is not limited to this.

[0279] Step 1404: Determine an arrangement order of the N objects according to the object relationship scores of the N objects in the first image.

[0280] In implementation, the arrangement order of the N objects is determined according to the object relationship scores of the N objects.

[0281] Step 1405: According to the arrangement order of the N objects, the search results of some or all of the N objects are fed back.

[0282] In an implementation, the search object may be determined from the N objects. In one implementation, the search object includes some of the N objects. In another embodiment, the search object includes some of the N objects.

[0283] As an example, a preset number of objects ranked high in the order of the N objects may be selected as search objects from the N objects according to the order of arrangement of the N objects.

[0284] As another example, the search object may be determined based on the object relationship score. When the object relationship score is large, it means that the corresponding object is more important and may be an object of interest to the user. Conversely, when the object relationship score is small, it means that the corresponding object is not important, that is, it may not be an object of interest to the user. Therefore, the electronic device may filter out objects with object relationship scores lower than the score threshold, and determine objects with object relationship scores higher than the score threshold as search objects.

[0285] The scoring threshold may be set by the user according to actual needs, or may be set by default by the electronic device, which is not limited in the embodiments of the present application.

[0286] It is worth mentioning that the embodiment of the present application filters the search objects from the first image according to the object relationship scores of the objects in the first image, so as to perform intention search only on the more important objects in the first image, thereby providing users with search results of search objects of interest, which can improve user experience.

[0287] After that, the search object is searched for intent, the search results are obtained, and the search results of the search object are fed back. For specific implementation, please refer to Fig. 9 Relevant content of step 905 in the illustrated embodiment.

[0288] Exemplarily, assuming that the search objects in the first image include face B and mobile phone A, the search results obtained include search results for face B and search results for mobile phone A. If the object relationship score of face B is greater than the object relationship score of mobile phone A, that is, the arrangement order of face B is before the arrangement order of mobile phone A, then in the process of displaying the search results, the search results of face B will be given priority. For example, the search results of face B can be displayed before the search results of mobile phone A.

[0289] In an embodiment of the present application, by determining the position importance values ​​of N objects in the first image in the two-dimensional space, and determining the degree of affiliation between each object and the reference object, and then determining the object relationship score of each object based on the position importance value and the degree of affiliation value, the search intention of the object is associated with the object attribute, thereby giving priority to recommending search results of search objects that the user may be interested in, thereby improving the user experience.

[0290] Please refer to Fig.17 , Fig.17 This is a flow chart of a method for feeding back search results according to an exemplary embodiment. Here, the method is described by taking the relevant process of determining the arrangement order of objects involved in the method as being implemented by the first processing unit alone as an example. Specifically, it may include part or all of the following contents:

[0291] The specific implementation of step 1701 to step 1702 can refer to the above Fig. 9 Step 901 to step 902 in the embodiment.

[0292] Step 1703: Perform scene recognition on the first image.

[0293] As an example, the electronic device performs scene recognition on the first image through the first processing unit to determine the scene category corresponding to the first image. As an example of the present application, the scene category may include but is not limited to character posters, bars, restaurants, coffee houses, cinemas, conference rooms, outdoor scenery, and outdoor landmarks.

[0294] In a possible implementation of the present application, the first processing unit may perform scene recognition on the first image through a scene recognition model, and the scene recognition model may be a pre-trained model that can determine the corresponding scene category based on any image. In implementation, the first image may be input into the scene recognition model, and the scene recognition model may perform scene recognition on the first image and output the scene category corresponding to the first image. For example, the scene recognition model determines that the scene category corresponding to the first image is a character poster.

[0295] As an example, the training process of the scene recognition model may include: obtaining a scene training sample set, the scene training sample set includes multiple scene training samples, each scene training sample includes a scene image marked with a scene category label. The scene training sample set is input into the model to be trained for training, and the training ends when the preset training conditions are met, so that the scene recognition model can be obtained.

[0296] The preset training condition can be set according to actual needs. For example, the preset training condition can refer to the number of training times reaching a number threshold, or the preset training condition can also refer to the scene recognition accuracy of the trained model reaching an accuracy threshold.

[0297] The number threshold may be set by the user according to actual needs, or may be set by default by the electronic device, and the embodiments of the present application are not limited to this.

[0298] The accuracy threshold may be set by the user according to actual needs, or may be set by default by the electronic device, which is not limited in the embodiments of the present application.

[0299] As an example, the model to be trained can be a classification network model. For example, the classification network model can be a model combining mobilenetV3 and SSD, that is, a model using mobilenetV3 as the network basis and the structure of SSD. Of course, here, the model to be trained is only a model combining mobilenetV3 and SSD as an example. In another embodiment, the model to be trained can also be other classification network models, and the embodiment of the present application does not limit this.

[0300] It should be noted that the training of the scene recognition model can be performed by the electronic device, or can also be performed by other devices, and then the trained scene recognition module is stored in the first processing unit of the electronic device.

[0301] Step 1704: Determine an arrangement order of the N objects in the first image according to the scene category of the first image and the object categories of the N objects in the first image.

[0302] Among them, different scenes correspond to at least one scene intention weight. The scene intention weight of any object is used to indicate the probability of any object being searched in the scene of the first image. The at least one scene intention weight corresponding to each scene can be pre-set according to actual needs. Exemplarily, the at least one scene intention weight corresponding to different scenes is shown in Table 2:

[0303] Table 2

[0304] Scene Category Scene intention weight 1 Scene Intent Weight 2 Scene Intent Weight 3 Scene intention weight n Character poster Celebrity 1.0 Shopping 0.9 Object recognition 0.8 ...... landscape Landmark 1.0 Object recognition 0.6 Celebrity 0.5 ...... Home Interior Shopping 1.0 Object recognition 0.6 Celebrity 0.6 ......

[0305] Each search intent corresponds to an object category. The electronic device can determine at least one corresponding scene intent weight according to the scene category. In addition, the corresponding search intent is determined according to the object categories of the N objects in the first image, so that the scene intent weight of each of the N objects can be determined. For example, assuming that the electronic device determines that the scene category of the first image is "character poster", the N objects in the first image include faces, mobile phones, and clothes, and the object categories of the N objects include: characters, commodities, and items. The corresponding search intents are celebrities, shopping, and object recognition, respectively. According to the above Table 1, it can be determined that the scene intent weights of each object are 1.0, 0.9, and 0.8, respectively.

[0306] Afterwards, the arrangement order of the N objects can be determined according to the scene intention weights of the N objects.

[0307] Step 1705: According to the arrangement order of the N objects, the search results of some or all of the N objects are fed back.

[0308] In one embodiment, the electronic device filters out objects having scene intention weights greater than a weight threshold from the first image, and determines the filtered objects as search objects.

[0309] Among them, the weight threshold can be set by the user according to actual needs, or can also be set by default by the electronic device, and the embodiments of the present application are not limited to this.

[0310] Afterwards, the search object in the first image is searched for intent, search results are obtained, and the search results of the search object are fed back. For specific implementation, please refer to Fig. 9 Relevant content of step 905 in the illustrated embodiment.

[0311] Exemplarily, the electronic device can sort the search intent according to the scene intent weight of each search object. For example, assuming that the search objects include face B and mobile phone A, the sorting results are: celebrities, shopping. That is, the most likely search intent corresponding to the first image is celebrity search, followed by shopping, such as purchasing the mobile phone in the first image. Since the scene intent weight of celebrities is higher than the scene intent weight of shopping, that is, the arrangement order of face B is before the arrangement order of mobile phone A, in the process of feedback of search results, the search results of face B are fed back first, for example, the search results of face B can be displayed before the search results of mobile phone A.

[0312] In the embodiment of the present application, the scene category of the first image is determined, and then the search intent ranking of the search object in the first image is determined according to the scene category, and the search results of the search object in the first image are displayed according to the search intent ranking. In this way, the search results are combined with the scene, so that the search results are closer to the scene, and the search results are prevented from being separated from the scene of the first image, so that the search results displayed to the user are more consistent with the scene of the first image, so as to give priority to recommending search results that the user may be interested in, thereby improving the user experience.

[0313] It should be noted that the above-mentioned embodiments are described by taking the case where the relevant process of determining the arrangement order of objects involved in the method is independently implemented by a processing unit as an example. As mentioned above, in one embodiment, the relevant process of determining the arrangement order of objects involved in the method provided by the embodiment of the present application can be implemented by any two of the above-mentioned three processing units in combination, or can also be implemented by the above-mentioned three processing units in combination with each other. Next, the specific implementation of the method provided by the embodiment of the present application is introduced by taking the implementation by the combination of three processing units as an example:

[0314] Please refer to Fig.18 , Fig.18 This is a schematic flow chart of a search result feedback method provided in an embodiment of the present application. The method may include part or all of the following contents:

[0315] Steps 1801 to 1802 can refer to the above Fig. 9 Step 901 to step 902 in the illustrated embodiment.

[0316] Step 1803: Determine the scene intention weight of each object in the N objects through the first processing unit.

[0317] The scene intention weight is used to indicate the probability of the first object being searched in the scene of the first image, where the first object is any one of the N objects. Fig. 9 The embodiment shown.

[0318] Step 1804: Determine the confidence score of each of the N objects by the second processing unit.

[0319] The confidence score is the similarity between the first object and the images in the image library. For its specific implementation, please refer to Fig.14 The embodiment shown.

[0320] Step 1805: Determine the object relationship score of each of the N objects by the third processing unit.

[0321] The object relationship score is used to indicate the importance of the first object in the first image. For example, the object relationship score is determined based on the position importance value, or based on the position importance value and the attachment relationship value. For specific implementation, please refer to Fig.17 The embodiment shown.

[0322] It should be noted that there is no sequential execution order for the first processing unit, the second processing unit and the third processing unit in the above steps. In one embodiment, the first processing unit, the second processing unit and the third processing unit can be executed in parallel.

[0323] Step 1806: Determine the priority of the first object based on the confidence score, object relationship score and scene intention weight of the first object.

[0324] The priority of the first object corresponds to the sorting order of the first object, that is, the higher the priority of the first object, the higher the sorting order.

[0325] The first object is any one of the N objects, that is, the first object is taken as an example for explanation here. In implementation, the priority of each object is determined based on the confidence score, object relationship score and scene intention weight of each object.

[0326] For example, assuming that the object relationship score is determined based on the position importance value and the dependency value, S represents the scene intention weight, F represents the confidence score of the object, L and H represent the position importance value and the dependency value of the object respectively, then the priority value of each object can be determined by the following formula (2), and the priority value is used to indicate the priority:

[0327] f n =xF n +yL n +kH n +zS n-c-I (2)

[0328] Among them, f n Indicates the priority value of object n, F n represents the confidence score of object n, L n Indicates the position importance value of object n, H n Represents the degree of affiliation of object n, S n-c-I The scene intention weight corresponding to the object category of object n in the first image. x, y, k, z are preset values ​​respectively.

[0329] Step 1807: Determine the search object from the first image based on the priorities of the N objects.

[0330] The search object includes a part of or all of the N objects included in the first image.

[0331] Step 1808: Obtain search results for the search object.

[0332] Step 1809: Feedback the search results of the search object.

[0333] In an embodiment of the present application, in the process of performing intent search based on a first image, the scene category of the first image is determined, the confidence score and the object relationship score of the object in the first image are determined, and then the priority of the object is determined based on the scene intention weight corresponding to the scene category, and the confidence score and the object relationship score of the object in the first image. The objects in the first image are sorted and displayed according to the priority, so that the search results of objects that the user may be interested in can be displayed as much as possible, thereby improving the accuracy of the intent search and thereby improving the user experience.

[0334] It should be understood that the serial numbers of the steps in the above embodiments do not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0335] Corresponding to the method for feeding back search results described in the above embodiment, Fig.19This is a structural block diagram of a device for feeding back search results provided in an embodiment of the present application. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0336] Reference Fig.19 , the device comprises:

[0337] An acquisition module 1910 is configured to acquire a first image, where the first image includes M objects, where M is an integer greater than or equal to 2;

[0338] A determination module 1920, configured to determine, for N objects among the M objects, an arrangement order of the N objects when N is greater than or equal to 2, where N is a positive integer less than or equal to M;

[0339] The arrangement order of any one of the N objects is determined based on any one or more of a scene intention weight, a confidence score, and an object relationship score, wherein the scene intention weight is used to indicate the probability of any one of the objects being searched in the scene corresponding to the first image, the confidence score is the similarity between any one of the objects and the images in the image library, and the object relationship score is used to indicate the importance of any one of the objects in the first image;

[0340] The feedback module 1930 is configured to feed back search results of some or all of the N objects according to the arrangement order of the N objects.

[0341] As an example of the present application, the determining module 1920 is used to:

[0342] Performing image detection processing on the first image by using a target detection model to obtain an object region of a first object in the first image and / or an object category of the first object, wherein the first object is any one of the N objects;

[0343] A confidence score of the first object is determined based on an object region of the first object in the first image and an object category of the first object, and / or an object relationship score of the first object is determined based on the object region of the first object in the first image, and / or a scene intention weight of the first object is determined based on the object category of the first object.

[0344] As an example of the present application, the determining module 1920 is used to:

[0345] determining a library of images corresponding to an object category of the first object;

[0346] determining a similarity between an object region of the first object in the first image and each of a plurality of images included in the image library;

[0347] The maximum similarity among the determined multiple similarities is used as the confidence score of the first object.

[0348] As an example of the present application, the first image includes a plurality of regions, and each of the plurality of regions has a preset score for indicating the importance of the location of each region, and the preset scores of at least two of the plurality of regions are different; the determining module 1920 is used to:

[0349] determining a position importance value of the first object in the first image based on a preset score of each area included in the object area of ​​the first object in the first image;

[0350] An object relationship score of the first object is determined based on the position importance value of the first object.

[0351] As an example of the present application, the determining module 1920 is used to:

[0352] When the first image includes a reference object, based on an object region of the first object in the first image and an object region of the reference object in the first image, obtaining a distance between the first object and the reference object in the first image as a dependency degree value of the first object;

[0353] An object relationship score of the first object is determined based on the position importance value and the dependency relationship value of the first object.

[0354] As an example of the present application, the determining module 1920 is used to:

[0355] determining a scene category of the first image;

[0356] The scene intent weight of the first object is determined based on the scene category of the first image and the object category of the first object, as well as the correspondence between the scene category, the object category and the scene intent weight.

[0357] As an example of the present application, the quality scores of the N objects in the first image are greater than or equal to a quality score threshold, and the quality score of any one of the objects is determined based on the blurriness and / or completeness of the any one of the objects.

[0358] As an example of the present application, the feedback module 1930 is used to:

[0359] Obtain object tags included in search results of some or all of the N objects;

[0360] Based on the acquired object tags, query information corresponding to some or all of the N objects is generated, where the query information is used to prompt whether information associated with the object tags needs to be acquired;

[0361] According to the arrangement order of the N objects, query information of some or all of the N objects is displayed to feed back search results of some or all of the N objects.

[0362] As an example of the present application, the N objects include a second object and a third object, the arrangement order of the second object is before the arrangement order of the third object, and the display order of the search results corresponding to the second object on the display screen is before the display order of the search results corresponding to the third object on the display screen.

[0363] In an embodiment of the present application, a first image is obtained, and the first image includes M objects, where M is an integer greater than or equal to 2. For N objects among the M objects, when the number of N objects is multiple, the arrangement order of the N objects is determined, and according to the arrangement order of the N objects, the search results of some or all of the N objects are fed back. Among them, the arrangement order of any object among the N objects is determined based on any one or more of the scene intention weight, the confidence score, and the object relationship score. The scene intention weight is used to indicate the probability of any object being searched in the scene corresponding to the first image, the confidence score is the similarity between any object and the image in the image library, and the object relationship score is used to indicate the importance of any object in the first image. In this way, the N objects are screened and sorted according to the arrangement order of the N objects, so that the search results of objects with good search results and that the user may be interested in are fed back to the user first, so that the feedback has a certain degree of pertinence, thereby improving the accuracy of the search results of the final feedback.

[0364] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0365] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0366] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0367] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the system embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0368] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0369] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0370] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to an electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electric carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0371] Finally, it should be noted that the above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for feeding back search results, characterized in that: The method comprises: Acquire a first image, where the first image includes M objects, where M is an integer greater than or equal to 2; For N objects among the M objects, when N is equal to 1, directly feeding back the search results of the N objects; when N is greater than or equal to 2, determining the arrangement order of the N objects, where N is a positive integer less than or equal to M; The arrangement order of any one of the N objects is determined based on any one or more of a scene intention weight, a confidence score, and an object relationship score, wherein the scene intention weight is used to indicate the probability of any one of the objects being searched in the scene corresponding to the first image, the confidence score is the similarity between any one of the objects and the images in the image library, and the object relationship score is used to indicate the importance of any one of the objects in the first image; According to the arrangement order of the N objects, search results of some or all of the N objects are fed back.

2. The method according to claim 1, characterized in that Any one or more of the scene intention weight, confidence score, and object relationship score of any one of the N objects is determined by: Performing image detection processing on the first image by using a target detection model to obtain an object region of a first object in the first image and / or an object category of the first object, wherein the first object is any one of the N objects; A confidence score of the first object is determined based on an object region of the first object in the first image and an object category of the first object, and / or an object relationship score of the first object is determined based on the object region of the first object in the first image, and / or a scene intention weight of the first object is determined based on the object category of the first object.

3. The method according to claim 2, characterized in that The determining a confidence score of the first object based on an object region of the first object in the first image and an object category of the first object comprises: determining a library of images corresponding to an object category of the first object; determining a similarity between an object region of the first object in the first image and each of a plurality of images included in the image library; The maximum similarity among the determined multiple similarities is used as the confidence score of the first object.

4. The method according to claim 2, characterized in that The first image includes a plurality of regions, and each of the plurality of regions has a preset score indicating a position importance of each region, and the preset scores of at least two of the plurality of regions are different; The determining the object relationship score of the first object based on the object region of the first object in the first image comprises: determining a position importance value of the first object in the first image based on a preset score of each area included in the object area of ​​the first object in the first image; An object relationship score of the first object is determined based on the position importance value of the first object.

5. The method according to claim 4, characterized in that Before determining the object relationship score of the first object based on the position importance value of the first object, the method further includes: When the first image includes a reference object, based on an object region of the first object in the first image and an object region of the reference object in the first image, obtaining a distance between the first object and the reference object in the first image as a dependency degree value of the first object; The determining the object relationship score of the first object based on the position importance value of the first object includes: An object relationship score of the first object is determined based on the position importance value and the dependency relationship value of the first object.

6. The method according to claim 2, characterized in that The determining the scene intention weight of the first object based on the object category of the first object includes: determining a scene category of the first image; The scene intent weight of the first object is determined based on the scene category of the first image and the object category of the first object, as well as the correspondence between the scene category, the object category and the scene intent weight.

7. The method according to any one of claims 1 to 6, characterized in that The quality scores of the N objects in the first image are greater than or equal to a quality score threshold, and the quality score of any one of the objects is determined based on the blurriness and / or completeness of the any one of the objects.

8. The method according to any one of claims 1 to 6, characterized in that Feedback of search results of some or all of the N objects according to the arrangement order of the N objects includes: Obtain object tags included in search results of some or all of the N objects; Based on the acquired object tags, query information corresponding to some or all of the N objects is generated, where the query information is used to prompt whether information associated with the object tags needs to be acquired; According to the arrangement order of the N objects, query information of some or all of the N objects is displayed to feed back search results of some or all of the N objects.

9. The method as claimed in claim 7, characterized in that Feedback of search results of some or all of the N objects according to the arrangement order of the N objects includes: Obtain object tags included in search results of some or all of the N objects; Based on the acquired object tags, query information corresponding to some or all of the N objects is generated, where the query information is used to prompt whether information associated with the object tags needs to be acquired; According to the arrangement order of the N objects, query information of some or all of the N objects is displayed to feed back search results of some or all of the N objects.

10. The method according to any one of claims 1 to 6, characterized in that The N objects include a second object and a third object, the arrangement order of the second object is before the arrangement order of the third object, and the display order of the search results corresponding to the second object on the display screen is before the display order of the search results corresponding to the third object on the display screen.

11. The method according to claim 7, characterized in that The N objects include a second object and a third object, the arrangement order of the second object is before the arrangement order of the third object, and the display order of the search results corresponding to the second object on the display screen is before the display order of the search results corresponding to the third object on the display screen.

12. The method of claim 8, wherein: The N objects include a second object and a third object, the arrangement order of the second object is before the arrangement order of the third object, and the display order of the search results corresponding to the second object on the display screen is before the display order of the search results corresponding to the third object on the display screen.

13. A device for feeding back search results, characterized in that: The device comprises: An acquisition module, configured to acquire a first image, wherein the first image includes M objects, where M is an integer greater than or equal to 2; a determination module, configured to, for N objects among the M objects, directly feed back the search results of the N objects when N is equal to 1; and determine the arrangement order of the N objects when N is greater than or equal to 2, where N is a positive integer less than or equal to M; The arrangement order of any one of the N objects is determined based on any one or more of a scene intention weight, a confidence score, and an object relationship score, wherein the scene intention weight is used to indicate the probability of any one of the objects being searched in the scene corresponding to the first image, the confidence score is the similarity between any one of the objects and the images in the image library, and the object relationship score is used to indicate the importance of any one of the objects in the first image; A feedback module is used to feed back search results of some or all of the N objects according to the arrangement order of the N objects.

14. The device according to claim 13, characterized in that The determination module is used for: Performing image detection processing on the first image by using a target detection model to obtain an object region of a first object in the first image and / or an object category of the first object, wherein the first object is any one of the N objects; A confidence score of the first object is determined based on an object region of the first object in the first image and an object category of the first object, and / or an object relationship score of the first object is determined based on the object region of the first object in the first image, and / or a scene intention weight of the first object is determined based on the object category of the first object.

15. The device according to claim 14, characterized in that The determination module is used for: determining a library of images corresponding to an object category of the first object; determining a similarity between an object region of the first object in the first image and each of a plurality of images included in the image library; The maximum similarity among the determined multiple similarities is used as the confidence score of the first object.

16. The device according to claim 14, characterized in that The first image includes a plurality of regions, and each of the plurality of regions has a preset score indicating the importance of the location of each region, and the preset scores of at least two of the plurality of regions are different; the determining module is used to: determining a position importance value of the first object in the first image based on a preset score of each area included in the object area of ​​the first object in the first image; An object relationship score of the first object is determined based on the position importance value of the first object.

17. The device according to claim 16, characterized in that The determination module is used for: When the first image includes a reference object, based on an object region of the first object in the first image and an object region of the reference object in the first image, obtaining a distance between the first object and the reference object in the first image as a dependency degree value of the first object; An object relationship score of the first object is determined based on the position importance value and the dependency relationship value of the first object.

18. A device for feeding back search results, characterized in that: The device includes a memory and a processor; The memory is used to store a program that supports the device to execute the method described in any one of claims 1 to 12, and to store data involved in implementing the method described in any one of claims 1 to 12; The processor is configured to execute the program stored in the memory.

19. A computer-readable storage medium, wherein instructions are stored in the computer-readable storage medium, characterized in that: When the method is executed on a computer, the computer is enabled to execute the method according to any one of claims 1 to 12.

Images