Application search method, device and medium

Abstract

The present application provides an application search method, device and medium, which can efficiently and accurately search for APP without entering search keywords. The method includes: obtaining an image of the application to be searched, the image including the icon of the application to be searched; extracting the constituent elements of the image of the application to be searched, the constituent elements including the graphic elements and color elements of the icon; searching for an application corresponding to the application to be searched from an application index library based on the constituent elements, wherein the application index library contains constituent elements of the images of multiple applications.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and more specifically, to an application search method, an electronic device, and a computer-readable storage medium. Background Art

[0002] With the popularity of smart phones and the increasing maturity of mobile Internet, a large number of mobile applications (APPs) have made our daily lives more convenient, such as map APPs, shopping APPs, food rating APPs, etc.

[0003] Existing APP search methods are mainly based on keyword search, that is, users input search keywords, and a list of APP search results is output according to the APP name or tag and relevance. However, in actual applications, there are scenarios where users cannot perform text search, such as the following scenarios: (1) users have reading and writing disabilities and cannot input search keywords; (2) users cannot input the local language during cross-border travel but need to download local APPs; (3) users want to search APPs quickly and find it inconvenient to manually input search keywords. Therefore, it is necessary to provide an efficient and accurate APP search method that does not require the input of search keywords. Summary of the invention

[0004] The present application provides an application search method, an electronic device, and a computer-readable storage medium, which can efficiently and accurately search for APPs without entering search keywords.

[0005] In a first aspect, an embodiment of the present application provides an application search method, characterized in that it includes: obtaining an image of an application to be searched, the image including an icon of the application to be searched; extracting constituent elements of the image of the application to be searched, the constituent elements including graphic elements and color elements of the icon; searching for an application corresponding to the application to be searched from an application index library based on the constituent elements, wherein the application index library contains the constituent elements of images of multiple applications.

[0006] According to the technical solution in the embodiments of the present application, on the one hand, since applications are searched by image processing, applications can be efficiently searched without entering search keywords; on the other hand, since the search is performed in combination with multiple component elements of the image of the application to be searched, the accuracy of the application search results can be improved.

[0007] In a possible implementation manner, the component elements further include text elements, and the text elements include text elements in the icon and / or text elements below the icon.

[0008] By extracting text elements from the image of the application to be searched and searching for the application in combination with the text elements, the accuracy of the application search results can be further improved.

[0009] In one possible implementation, searching for an application corresponding to the application to be searched from an application index library based on the constituent elements includes: determining element similarities between each constituent element of the application to be searched and corresponding constituent elements of each application in the application index library, the element similarity including one or more of graphic similarity, color similarity and text similarity; and determining the application corresponding to the application to be searched based on the element similarity.

[0010] In one possible implementation, the determining of the application corresponding to the application to be searched based on the element similarity includes: determining multiple application sets corresponding to the application to be searched based on the element similarity, each application in the application set is at least one application whose element similarity is greater than a predetermined threshold; determining the union of each of the application sets to obtain multiple candidate applications; determining the program confidence of each candidate application based on the element similarity, the program confidence indicating the probability that the candidate application is similar to the application to be searched; and determining the application corresponding to the application to be searched based on the size of the program confidence.

[0011] In one possible implementation, the application index library also includes: the weight of the corresponding elements of the candidate applications, and the determination of the program confidence between each candidate application and the application to be searched based on the element similarity includes: determining the weight of each constituent element of the application to be searched and the weight of the corresponding element of the candidate application; determining the element confidence of the corresponding element of the candidate application based on the weight of each constituent element of the application to be searched, the weight of the corresponding element of the candidate application and the corresponding element similarity, the element confidence indicating the probability that the corresponding element belongs to the element of the application to be searched; determining the program confidence of each candidate application based on the element confidence.

[0012] By setting weights of constituent elements in an application index library, the accuracy of application search can be improved, so that the application can be accurately identified when the application undergoes design deformation.

[0013] In one possible implementation, determining the element confidence of the corresponding element of the candidate application includes: inputting the weights of the constituent elements of the application to be searched, the weights of the corresponding constituent elements of the candidate application, and the corresponding element similarities into an element scoring model, wherein the element scoring model is a neural network model; and determining the element confidence of the corresponding element of the candidate application through the element scoring model.

[0014] In one possible implementation, the program confidence of each candidate application is determined based on the element confidence, including: performing a weighted sum operation on the element confidence of each component element under each category of component elements of the candidate application to determine the category element confidence of the candidate application, the category element confidence indicating the probability that a category of component elements of the candidate application all belong to the application to be searched; based on the category element confidence, determining the program confidence of the candidate application.

[0015] In one possible implementation, the category element confidence of the candidate application is determined based on the category element confidence, including: inputting the category element confidence of each category of component elements into a category scoring model, wherein the category scoring model is a neural network model; and determining the program confidence of each candidate application through the category scoring model.

[0016] In a possible implementation, extracting component elements of the image of the application to be searched includes: extracting graphic elements of the image of the application to be searched by using a semantic segmentation model; and extracting color elements of the image of the application to be searched by using a color quantization model.

[0017] In a possible implementation, the constituent elements include text elements, and extracting the constituent elements of the image of the application to be searched further includes: extracting the text elements of the image of the application to be searched by using a character recognition model.

[0018] In a possible implementation, the application index library further includes: element feature indexes and element features corresponding to various component elements of the application.

[0019] In a second aspect, an electronic device is provided, comprising: an image acquisition unit, for acquiring an image of an application to be searched, the image comprising an icon of the application to be searched; a processing unit, for extracting constituent elements of the image of the application to be searched, the constituent elements comprising graphic elements and color elements of the icon; and searching an application corresponding to the application to be searched from an application index library based on the constituent elements, wherein the application index library contains the constituent elements of images of multiple applications.

[0020] In a possible implementation manner, the component elements further include text elements, and the text elements include text elements in the icon and / or text elements below the icon.

[0021] In one possible implementation, the processing unit is used to: determine the element similarity between each component element of the application to be searched and the corresponding component elements of each application in the application index library, the element similarity including one or more of graphic similarity, color similarity and text similarity; determine the application corresponding to the application to be searched based on the element similarity.

[0022] In one possible implementation, the processing unit is used to include: determining multiple application sets corresponding to the application to be searched based on the element similarity, each application in the application set is at least one application whose element similarity is greater than a predetermined threshold; determining the union of each of the application sets to obtain multiple candidate applications; determining the program confidence of each candidate application based on the element similarity, the program confidence indicating the probability that the candidate application is similar to the application to be searched; and determining the application corresponding to the application to be searched based on the size of the program confidence.

[0023] In one possible implementation, the application index library also includes: the weight of the corresponding element of the candidate application, and the processing unit is used to: determine the weight of each component element of the application to be searched and the weight of the corresponding element of the candidate application; determine the element confidence of the corresponding element of the candidate application based on the weight of each component element of the application to be searched, the weight of the corresponding element of the candidate application and the corresponding element similarity, the element confidence represents the probability that the corresponding element belongs to the element of the application to be searched; based on the element confidence, determine the program confidence of each candidate application.

[0024] In one possible implementation, the processing unit is used to: input the weights of the constituent elements of the application to be searched, the weights of the corresponding constituent elements of the candidate application, and the corresponding element similarities into an element scoring model, wherein the element scoring model is a neural network model; and determine the element confidence of the corresponding element of the candidate application through the element scoring model.

[0025] In one possible implementation, based on the element confidence, the processing unit is used to: perform a weighted sum operation on the element confidence of each component element under each category of component elements of the candidate application to determine the category element confidence of the candidate application, wherein the category element confidence indicates the probability that a category of component elements of the candidate application all belong to the application to be searched; and determine the program confidence of the candidate application based on the category element confidence.

[0026] In a possible implementation, the processing unit is used to: input the category element confidence of each category component element into a category scoring model, where the category scoring model is a neural network model; and determine the program confidence of each candidate application through the category scoring model.

[0027] In a possible implementation, the processing unit is used to: extract graphic elements of the image of the application to be searched by using a semantic segmentation model; and extract color elements of the image of the application to be searched by using a color quantization model.

[0028] In a possible implementation, the component elements include text elements, and the processing unit is used to extract the text elements of the image of the application to be searched by using a character recognition model.

[0029] In a possible implementation, the application index library further includes: element feature indexes and element features corresponding to various component elements of the application.

[0030] In a third aspect, an embodiment of the present application further provides a computer-readable storage medium for storing a computer program, wherein the computer program includes instructions for implementing the method in the above-mentioned first aspect or any possible implementation thereof.

[0031] In a fourth aspect, an embodiment of the present application further provides a computer program product, which includes instructions. When the instructions are executed on a computer, the computer implements the method in the first aspect or any possible implementation thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is a schematic diagram of the structure of an electronic device 100 provided in an embodiment of the present application;

[0033] Figure 2 is a software structure block diagram of the electronic device provided in the embodiment of the present application;

[0034] Figure 3 is a flowchart of an application search method provided by some embodiments of the present application;

[0035] Figure 4 is a schematic diagram of a display interface including application program icons provided in some embodiments of the present application;

[0036] Figure 5 is a flowchart of an application search method provided by other embodiments of the present application;

[0037] Figure 6 is a schematic diagram of an image containing an application icon provided by some embodiments of the present application;

[0038] Figure 7 Some embodiments of the present application provide Figure 6 Schematic diagram of an image to be cropped;

[0039] FIG. 8 is provided by some embodiments of the present application Figure 7 A schematic diagram of a graphical element of an image of an application;

[0040] Fig. 9 Some embodiments of the present application provide detection Figure 7 A schematic diagram of text elements in an image of an application;

[0041] Fig.10 Some embodiments of the present application provide Figure 7 A schematic diagram of color elements of an image in an application;

[0042] Fig.11 is a schematic diagram of an image of a candidate application provided by some embodiments of the present application;

[0043] Fig.12 is a schematic diagram of an initial BP neural network model structure provided by some embodiments of the present application;

[0044] Fig.13 is a schematic diagram of an application index library provided by some embodiments of the present application;

[0045] Fig.14 is a schematic diagram of an electronic device provided in some embodiments of the present application. DETAILED DESCRIPTION

[0046] Before introducing the method embodiments of the present application, the technologies involved in the embodiments of the present application are described as follows:

[0047] Composition elements: elements contained in the image of the application, including graphic elements, text elements, and color elements.

[0048] Element similarity: represents the degree of similarity between the constituent elements of the image of the application. For example, the similarity of graphic elements can be represented by the cosine value of the angle between the feature vectors of two graphic elements.

[0049] Element confidence: indicates the probability that the constituent elements of the application in the application index library belong to the elements of the application to be searched.

[0050] Category element confidence: It indicates the probability that all the elements of a category of candidate applications belong to the elements of the application to be searched.

[0051] Program confidence: Indicates the probability that the candidate application is similar to the search application.

[0052] Weight of a component element: indicates the importance of the component element in the application's icon. For example, the element weight may be the proportion of the element in the application's image. For example, the weight of a graphic element may be the ratio of the area occupied by the graphic element to the area of ​​the application's icon. The weight of a text element may be the ratio of the area occupied by the text element to the area of ​​the application's icon. The weight of a color element may be the ratio of the area occupied by the color element to the area of ​​the application's icon.

[0053] In one technical solution, the feature vector corresponding to the APP icon is determined, and the application information corresponding to the feature vector is searched in the feature vector database. However, when the APP icon is relatively complex, the recognition accuracy of this technical solution is low and it is difficult to meet application requirements.

[0054] Based on the above content, the embodiment of the present application provides an application search method, which can be applied to electronic devices with image processing devices, such as smart phones, tablet computers, wearable electronic devices, etc. Through the application search method, the electronic device can accurately search for APPs in more complex scenarios without entering search keywords.

[0055] The implementation of the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0056] First, an electronic device having an image processing device, a user interface for such an electronic device, and an embodiment for using such an electronic device are introduced. In some embodiments, the electronic device may be a portable electronic device that also includes other functions such as a personal digital assistant and / or a music player function, such as a mobile phone, a tablet computer, a wearable electronic device with a wireless communication function (such as a smart watch), etc. Exemplary embodiments of portable electronic devices include but are not limited to portable electronic devices equipped with other operating systems. The above-mentioned portable electronic device may also be other portable electronic devices, such as a laptop computer (Laptop), etc. It should also be understood that in some other embodiments, the above-mentioned electronic device may not be a portable electronic device, but a desktop computer.

[0057] For example, Figure 1 1 shows a schematic diagram of the structure of the electronic device 100. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195.

[0058] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0059] The processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), a controller, a video codec, a digital signal processor (digital signal processor, DSP), a baseband processor, and / or a neural-network processing unit (neural-network processing unit, NPU), etc. Among them, different processing units may be independent components or integrated into one or more processors. In some embodiments, the electronic device 101 may also include one or more processors 110. Among them, the controller may generate an operation control signal according to the instruction opcode and the timing signal to complete the control of fetching and executing instructions. In some other embodiments, a memory may also be set in the processor 110 for storing instructions and data. Exemplarily, the memory in the processor 110 may be a high-speed cache memory. The memory may store instructions or data that the processor 110 has just used or circulated. If the processor 110 needs to use the instruction or data again, it can be directly called from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the electronic device 101 in processing data or executing instructions.

[0060] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a SIM card interface, and / or a USB interface, etc. Among them, the USB interface 130 is an interface that complies with the USB standard specification, and can specifically be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 101, and can also be used to transmit data between the electronic device 101 and a peripheral device. The USB interface 130 can also be used to connect headphones to play audio through the headphones.

[0061] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0062] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the electronic device 100. While the charging management module 140 is charging the battery 142, it may also power the electronic device through the power management module 141.

[0063] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0064] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0065] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0066] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0067] The wireless communication module 160 can provide wireless communication solutions for application in the electronic device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication technology (NFC), infrared technology (IR), etc.

[0068] Optionally, the wireless communication module 160 may be one or more devices integrating at least one communication processing module, wherein one communication processing module may correspond to one network interface, the network interface may be set in different service function modes, and the network interface set in different modes may establish a network connection corresponding to the mode.

[0069] For example, a network connection supporting the P2P function can be established through a network interface in the P2P function mode, a network connection supporting the STA function can be established through a network interface in the STA function mode, and a network connection supporting the AP function can be established through a network interface in the AP mode.

[0070] The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive signals to be sent from the processor 110, modulate and amplify them, and convert them into electromagnetic waves for radiation via the antenna 2.

[0071] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0072] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include one or more display screens 194.

[0073] In some embodiments of the present application, when the display panel is made of materials such as OLED, AMOLED, FLED, etc., the above Figure 1 The display screen 194 in the embodiment can be bent. Here, the display screen 194 can be bent means that the display screen can be bent to any angle at any position and can be maintained at the angle. For example, the display screen 194 can be folded in half from the middle to the left and right. It can also be folded in half from the middle to the top and bottom. In this application, a display screen that can be bent is referred to as a foldable display screen. The touch display screen can be a single screen or a display screen composed of multiple screens pieced together, which is not limited here.

[0074] The display screen 194 of the electronic device 100 can be a flexible screen. At present, the flexible screen has attracted much attention for its unique characteristics and huge potential. Compared with traditional screens, flexible screens have the characteristics of strong flexibility and bendability, which can provide users with a new interaction method based on bendable characteristics, and can meet users' more needs for electronic devices. For electronic devices equipped with a foldable display screen, the foldable display screen on the electronic device can be switched between a small screen in a folded form and a large screen in an unfolded form at any time. Therefore, users use the split-screen function on electronic devices equipped with a foldable display screen more and more frequently.

[0075] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0076] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0077] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include one or more cameras 193.

[0078] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0079] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0080] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission of business functions between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0081] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0082] The internal memory 121 can be used to store one or more computer programs, which include instructions. The processor 110 can enable the electronic device 101 to perform the screen-off display method provided in some embodiments of the present application, as well as various applications and data processing, by running the above instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system; the program storage area can also store one or more applications (such as a gallery, contacts, etc.). The data storage area can store data (such as photos, contacts, etc.) created during the use of the electronic device 101. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more disk storage components, a flash memory component, a universal flash storage (UFS), etc. In some embodiments, the processor 110 can enable the electronic device 101 to perform the screen-off display method provided in the embodiments of the present application, as well as other applications and data processing by running instructions stored in the internal memory 121, and / or instructions stored in a memory provided in the processor 110. The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0083] The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, and the like.

[0084] Among them, the pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 194, the electronic device 100 detects the touch operation intensity according to the pressure sensor 180A. The electronic device 100 can also calculate the touch position according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, an instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, an instruction to create a new short message is executed.

[0085] The gyro sensor 180B can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., X, Y, and Z axes) can be determined by the gyro sensor 180B. The gyro sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyro sensor 180B detects the angle of the electronic device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 100 through reverse movement to achieve anti-shake. The gyro sensor 180B can also be used for navigation and somatosensory game scenes.

[0086] The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device 100 in all directions (generally three axes). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the electronic device and is applied to applications such as horizontal and vertical screen switching and pedometers.

[0087] The ambient light sensor 180L is used to sense the ambient light brightness. The electronic device 100 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the electronic device 100 is in a pocket to prevent accidental touches.

[0088] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.

[0089] The temperature sensor 180J is used to detect temperature. In some embodiments, the electronic device 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold, the electronic device 100 reduces the performance of a processor located near the temperature sensor 180J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the electronic device 100 heats the battery 142 to avoid abnormal shutdown of the electronic device 100 due to low temperature. In other embodiments, when the temperature is lower than another threshold, the electronic device 100 performs a boost on the output voltage of the battery 142 to avoid abnormal shutdown caused by low temperature.

[0090] The touch sensor 180K is also called a "touch panel". The touch sensor 180K can be set on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the electronic device 100, which is different from the position of the display screen 194.

[0091] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The electronic device 100 may receive key input and generate key signal input related to user settings and function control of the electronic device 100.

[0092] Figure 2 1 is a software structure diagram of the electronic device 100 of the embodiment of the present application. The layered architecture divides the software into several layers, each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, namely, the application layer, the application framework layer, the Android runtime (Android runtime) and the system library, and the kernel layer. The application layer can include a series of application packages.

[0093] It should be noted that in the embodiment of the present application, only the Android operating system is used as an example for introduction. The electronic device 100 may also be equipped with other operating systems, such as: operating system, operating system, Operating system, etc., which is not limited in the embodiments of the present application.

[0094] like Figure 2 As shown, the application package may include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and other applications.

[0095] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0096] like Figure 2 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0097] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0098] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0099] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0100] The phone manager is used to provide communication functions of the electronic device 100, such as management of call status (including connecting, hanging up, etc.).

[0101] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0102] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0103] Android Runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0104] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0105] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0106] The system library may include multiple functional modules, such as surface manager, media libraries, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0107] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0108] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0109] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0110] A 2D graphics engine is a drawing engine for 2D drawings.

[0111] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0112] Figure 3 is a flowchart of an application search method provided by some embodiments of the present application. The method can be applied to Figure 1 In the electronic device 100 shown, the method may include steps S310 to S330, and the method is described in detail below with reference to the accompanying drawings.

[0113] In step S310, an image of the application to be searched is obtained, where the image includes an icon of the application to be searched.

[0114] In an exemplary embodiment, the image of the application to be searched may include the icon portion of the application to be searched and the text portion below the icon of the application, and the image of the application to be searched may be obtained by the camera 193 of the electronic device 100. For example, suppose user A wants to install an application on user B's mobile phone B on mobile phone A, for example, Figure 4 If APP1 is in the mobile phone A, user A can obtain the image of the application APP1 through the camera of mobile phone A. The image of the application APP1 may include the icon part of the application APP1 and the text part APP1 under the icon of the application APP1.

[0115] In step S320, component elements of the image of the application to be searched are extracted, and the component elements include graphic elements and color elements.

[0116] In an example embodiment, the graphic elements of the icon of the application to be searched are extracted through a semantic segmentation model. Specifically, the graphic elements in the icon of the application to be searched can be extracted through the image semantic segmentation model DeepLabV2, or the graphic elements in the icon of the application to be searched can be extracted using a deep separable convolution model. The graphic elements may include feature vectors of the graphic elements of the application to be searched, such as feature vectors of contour information of the graphic elements and high-level semantic feature vectors. It should be noted that the graphic elements in the icon of the application to be searched can also be extracted through other appropriate models, and the embodiments of the present application do not specifically limit this.

[0117] Furthermore, the color elements of the icon of the application to be searched can be extracted through a color quantization model. For example, the main colors in the icon of the application to be searched, such as the background color of the icon and the color of the graphics or text in the icon, can be extracted through the color quantization model octree quantization method. It should be noted that although the color quantization model is used for explanation, those skilled in the art should understand that the color elements of the icon of the application to be searched can also be extracted through other appropriate methods, such as extracting the color histogram of the icon through OPENCV, which is also within the protection scope of the embodiments of the present application.

[0118] In step S330, an application corresponding to the application to be searched is searched from an application index library based on the component elements, wherein the application index library contains component elements of images of multiple applications.

[0119] In an example embodiment, the element similarity between the feature vectors of each component element of the application to be searched and the feature vectors of the corresponding component elements of each application in the application index library is determined, and the target application corresponding to the application to be searched is determined based on the element similarity between each component element of the application to be searched and the corresponding component elements of each application. Specifically, the graphic similarity between the feature vectors of the graphic elements of the application to be searched and the feature vectors of the graphic elements of each application in the application index library can be determined; the color similarity between the feature vectors of the color elements of the application to be searched and the feature vectors of the color elements of each application in the application index library can be determined; and the icon corresponding to the application to be searched can be determined based on the graphic similarity and the color similarity.

[0120] Furthermore, in an exemplary embodiment, the constituent elements further include text elements, and the text elements include text elements in the icon of the application to be searched and / or text elements below the icon. Therefore, the text similarity between the feature vector of the text element of the application to be searched and the feature vector of the text element of each application in the application index library can be determined.

[0121] according to Figure 3 According to the technical solution in the example embodiment, constituent elements of the image of the application to be searched are extracted, the constituent elements including graphic elements and color elements, and a target application corresponding to the application to be searched is searched from the application index library based on the constituent elements. On the one hand, since the application is searched by image processing, the application can be searched efficiently without entering search keywords; on the other hand, since the search is performed in combination with multiple constituent elements of the image of the application to be searched, the accuracy of the application search results can be improved.

[0122] Figure 5 is a flowchart of an application search method provided by other embodiments of the present application. The application search method can be applied to Figure 1 The electronic device 100, the application search method is described in detail below with reference to the accompanying drawings.

[0123] Reference Figure 5 As shown, in step S510, an image containing an icon of the application to be searched is obtained.

[0124] In an exemplary embodiment, an image containing an icon of the application to be searched is acquired through a camera of the electronic device 100. Figure 6 As shown, the graphic includes icons of three applications, among which APP1 is the application to be searched.

[0125] In step S520, the icon of the application to be searched is detected and clipped.

[0126] In an example embodiment, the icon of the application to be searched is detected using a target detection model. For example, the icon of the application to be searched is detected using a candidate region-based target detection model to obtain an image of the application to be searched.

[0127] For example, refer to Figure 6 As shown in the figure, the target detection model based on candidate regions is used to detect the APP icon and the area where the text below the icon is located, and the detection result list ["box1":"[14,1000,142,175]","box2":"[360,1250,137,171]","box3":"[554,1007,136,178]"] is output. The first APP icon to be searched on the left is selected and cropped to obtain Figure 7 The APP image to be searched is shown.

[0128] It should be noted that the target detection model can be an R-CNN model or other appropriate target detection models such as a YOLO model or an SSD model, and the embodiments of the present application do not specifically limit this.

[0129] In step S530, the component elements of the image of the application are extracted, and the component elements include: graphic elements, text elements and color elements. Step S530 includes steps S532, S534 and S536. Step S530 is described in detail below.

[0130] In step S532, the graphic elements in the icon of the application to be searched are detected, and the feature vectors of the graphic elements are extracted.

[0131] In an example embodiment, the graphic elements of the icon of the application to be searched are extracted through a semantic segmentation model. Specifically, the graphic elements in the icon of the application to be searched can be extracted through the image semantic segmentation model DeepLabV2, or the graphic elements in the icon of the application to be searched can be extracted using a deep separable convolution model. The graphic elements may include feature vectors of the graphic elements of the application to be searched, such as feature vectors of contour information of the graphic elements and high-level semantic feature vectors. It should be noted that the graphic elements in the icon of the application to be searched can also be extracted through other appropriate models, and the embodiments of the present application do not specifically limit this.

[0132] For example, refer to Figure 7 As shown in the figure, the outline of the graphic elements of the APP icon to be searched is extracted through the semantic segmentation model. Figure 8a and Figure 8b, generate an image feature vector of the graphic element contour, and compress the generated image feature vector into 512 dimensions.

[0133] In step S534, the text element in the icon of the application to be searched is detected.

[0134] In an exemplary embodiment, OCR is used to identify text elements in the icon of the application to be searched and / or text elements under the icon, and the text elements may include one or more of words, letters, and numbers.

[0135] For example, OCR recognition technology can be used to identify the text elements in the icon of the APP to be searched and below the icon. Fig. 9 As shown, the following text elements are extracted from the APP icon to be searched through OCR recognition technology: [{"word":"618","bbox":[62,1018,46,21]},{"word":"XXX","bbox":[44,1097,82,22]}], and the APP name is extracted from the APP icon to be searched [{"word":"APP1","bbox":[35,1135,101,30]}].

[0136] In step S536, the color element in the icon of the application to be searched is detected.

[0137] In an example embodiment, the color elements of the icon of the application to be searched are extracted by a color quantization model. For example, the main colors in the icon of the application to be searched, such as the background color of the icon and the color of the graphics or text in the icon, can be extracted by an octree quantization method of the color quantization model.

[0138] For example, refer to Fig.10 As shown, the color elements of the icon of the application to be searched are extracted by the octree quantization method, and the extracted color elements are: [["rgb":[220,51,69],"percent":0.41], ["rgb":[220,51,69],"percent":0.14]], and converted to the Lab color space [["lab":[49.512,64.633,31.214],"percent":0.41], ["lab":[97.876,-0.345,0.946],"percent":0.14]].

[0139] It should be noted that although the color quantization model is used for explanation, those skilled in the art should understand that the color elements of the icon of the application to be searched can also be extracted by other appropriate methods, for example, the color histogram of the icon can be extracted by OPENCV, which is also within the scope of protection of the embodiments of the present application.

[0140] Next, in step S540, the corresponding application is searched from the application index library according to the constituent elements of the image of the application to be searched. Step S540 includes steps S542, S544 and S546. Step S540 is described in detail below.

[0141] In step S542, an application whose graphic similarity with the graphic element of the application to be searched is greater than a predetermined threshold is searched from the application index library.

[0142] In an example embodiment, the graphic similarity between the feature vector of the graphic element of the application to be searched and the feature vector of the graphic element of the application in the application index library may be determined, and the application having the graphic similarity greater than a predetermined threshold may be determined.

[0143] For example, let the feature vector of the graphic element of the application to be searched be A, and the feature vector of the graphic element of the application in the application index library be B. Then the similarity between A and B can be measured by cosine similarity, and the graphic similarity can be expressed by the following formula (1):

[0144]

[0145] Among them, Ai and Bi represent the components of vector A and vector B respectively.

[0146] The image similarity between A and B is determined by using the above (1) to obtain an application program whose image similarity is greater than a predetermined threshold. Figure 8a 79562 APPs were found based on the graphic elements in Figure 8b 13 apps were found by searching for graphic elements in .

[0147] In step S544, an application whose text similarity with the text elements of the application to be searched is greater than a predetermined threshold is searched from the application index library.

[0148] In an example embodiment, the text similarity between the character string of the text element of the application to be searched and the character string of the text element of the application in the application index library may be determined, and the application having the text similarity greater than a predetermined threshold may be determined.

[0149] For example, let the text element extracted from the name or icon of the application to be searched be a, and the text element of the application in the application index library be b. The text similarity between a and b can be measured by the Levenshtein distance, for example, it can be expressed as lev a,b (|a|,b|). Among them, |a|, |b| represent the length of two strings a and b. a,b (|a|,|b|) satisfies the constraint condition of the following formula (2).

[0150]

[0151] in, represents an indicator function, when a i =b i When a i ≠b i When , its value is 1. a,b (i,j) represents the distance between the first i characters of a and the first j characters of b (both i and j are subscripts starting from 1).

[0152] The above formula (2) is used to determine the text similarity between a and b, and to determine the applications whose text similarity is greater than a predetermined threshold. For example, 35,895 applications are searched based on the text element "618", 8 applications are searched based on the text element "XXX", and 22 applications are searched based on the application name "APP1".

[0153] In step S546, an application whose color similarity with the color element of the application to be searched is greater than a predetermined threshold is searched from the application index library.

[0154] In an example embodiment, the color similarity between the color elements of the application to be searched and the color elements of the applications in the application index library may be determined, and the applications having the color similarity greater than a predetermined threshold may be determined.

[0155] For example, suppose the color element of the application to be searched is denoted in the Lab color space as The color elements of the application searched in the application index library are recorded in the Lab color space as but and The similarity can be measured using the International Commission on Illumination It can be expressed by the following formula (3):

[0156]

[0157] Use the above formula (3) to determine and The similarity between the two colors is calculated to obtain the applications whose color similarity is greater than a predetermined threshold. For example, 1,589,532 applications are searched based on color element a, and 2,658,923 applications are searched based on color element b.

[0158] Next, in step S550, the search results corresponding to the various component elements searched in step S540 are sorted.

[0159] In an example embodiment, the search results are sorted in descending order of element similarity, a predetermined number of applications are selected from the sorted results to form multiple application sets, and the multiple application sets are unioned to generate a candidate application set.

[0160] For example, the search results of each component element are sorted according to the size of the element similarity, and after taking the top 80% of the applications, multiple application sets are obtained, and the union of the multiple application sets is taken to obtain a candidate application set of 2,127,138 APPs.

[0161] Furthermore, in an exemplary embodiment, a weighted operation may be performed on the element similarities of the constituent elements of the candidate applications in the candidate application set to obtain the program similarities between the candidate applications and the application to be searched, and the candidate applications may be sorted according to the program similarities.

[0162] In step S560, the APP search results are output.

[0163] In an exemplary embodiment, after the ranking results of each candidate application in the candidate application set are obtained, the APP search results are output on the user interface of the electronic device 100. For example, the top 10% of APPs in the candidate application set are output on the interface of the application store of the electronic device 100.

[0164] according to Figure 5 According to the technical solution in the example embodiment, on the one hand, since applications are searched by image processing, applications can be searched efficiently without entering search keywords; on the other hand, since the search is performed in combination with multiple component elements of the image of the application to be searched, the accuracy of the application search results can be improved.

[0165] Further, in the example embodiment, the program confidence of each candidate application is determined based on the element similarity, and the program confidence represents the probability that the candidate application is similar to the application to be searched; and the application corresponding to the application to be searched is determined based on the magnitude of the program confidence. Specifically, the application index library also includes: the weight of the corresponding element of the candidate application, and the program confidence between each candidate application and the application to be searched is determined based on the element similarity, including: determining the weight of each constituent element of the application to be searched and the weight of the corresponding element of the candidate application; determining the element confidence of the corresponding element of the candidate application based on the weight of each constituent element of the application to be searched, the weight of the corresponding element of the candidate application, and the corresponding element similarity, and the element confidence represents the probability that the corresponding element belongs to the element of the application to be searched; based on the element confidence, the program confidence of each candidate application is determined.

[0166] For example, let q represent the application to be searched, c represent an application in the candidate application list, and I q represents the weight of the component i in the icon of the application to be searched, I c represents the weight of the component i in the icon of the candidate application, S represents the similarity between the component i in the icon of q and the component i in the icon of c, then the element confidence of the component i in the icon of the candidate application can be calculated by the element scoring BP neural network model, and the element confidence f i Specifically, it can be expressed by the following formula (4):

[0167] f i (I q ,I c ,S) (4)

[0168] Element scoring BP neural network model construction: reference Fig.12 As shown, the initial BP neural network model is constructed. q ,I c The three parameters S, S, and S are input into the input layer of the initial BP neural network model, and whether the constituent element i of the candidate application belongs to the element of the application to be searched is used as the output of the BP neural network model. For example, the result of whether the constituent element i of the candidate application belongs to the element of the application to be searched can be one-hot encoded and used as the output of the BP neural network model. In addition, the initial BP neural network model can be trained by the applications in the application index library to obtain the trained BP neural network model as the element scoring BP neural network model in practical applications.

[0169] Furthermore, let a represent the graphic element, b represent the text element, and c represent the color element. Then, the program confidence of the candidate application can be obtained through the category scoring BP neural network model, and the program confidence G can be expressed by the following formula (5):

[0170] G(H a ,H b ,H c ) (5)

[0171] Wherein, H represents the confidence of each component element, and H can be expressed by the following formula (6):

[0172]

[0173] Where w is an empirical value, and I can be determined by the following formula (7):

[0174]

[0175] The construction method of category scoring BP neural network model: Fig.12 As shown, the confidence of the graphic element class, text element class, and color element class is input into the input layer of the initial BP neural network model, and whether the candidate application is the application to be searched is one-hot encoded as the output of the initial BP neural network model. In addition, the initial BP neural network model can be trained according to the applications in the application index library to obtain the trained BP neural network model as the category scoring neural network model in actual use.

[0176] For example, for Fig.11 For the candidate APPs shown, the following results were calculated using the element scoring BP neural network model: the confidence of graphic element a is 0.004, and the confidence of graphic element b is 0.784; the confidence of text element "618" is 0.000, the confidence of text element "HUAWEI" is 0.689, and the confidence of APP name "XXX" is "0.301"; the confidence of color element a is 0.002, and the confidence of color element b is 0.776.

[0177] Furthermore, the confidence of the category elements of the candidate APP can be calculated. Specifically, the confidence of the graphic element is 0.784, the confidence of the text element is 0.689, and the confidence of the color element is 0.012.

[0178] Furthermore, the program confidence of the candidate APP calculated by the category scoring BP neural network model is 0.988.

[0179] Next, the candidate APPs in the candidate APP set are sorted according to the program confidence level, and the sorted results are output.

[0180] According to the technical solution in the example embodiment of the present application, on the one hand, by setting the weights of the constituent elements in the application index library, the accuracy of application search can be improved, so that the APP can be accurately identified when the APP undergoes design deformation; on the other hand, by using a neural network model to determine the program confidence of the candidate APP, the application search can be further performed efficiently and accurately.

[0181] Fig.13 It is a schematic diagram of an application index library provided by some embodiments of the present application.

[0182] In an example embodiment, the application index library may be pre-built. Fig.13 As shown, each application in the application index library corresponds to three types of elements: graphic element class, text element class, and color element class. Each type of element includes an element feature index, an element feature (graphic feature vector, text string, color value), and a weight. Among them, the weight represents the importance of the element in the icon of the application. For example, the weight can be the proportion of the element in the image of the application. For example, the weight of the graphic element can be the ratio of the area occupied by the graphic element to the area of ​​the icon of the application. The weight of the text element can be the ratio of the area occupied by the text element to the area of ​​the icon of the application. The weight of the color element can be the ratio of the area occupied by the color element to the area of ​​the icon of the application. In addition, in some example embodiments, the element weight can also be other appropriate values. For example, the element weight can be the frequency of the element appearing in public visual materials such as the interface, posters, and advertisements of the application, which is also within the scope of protection of the present application.

[0183] By setting the weights of the constituent elements in the application index library, the accuracy of application search can be improved, so that the APP can be accurately identified when the design of the APP is deformed.

[0184] Fig.14 is a schematic diagram of an electronic device provided in some embodiments of the present application.

[0185] Reference Fig.14As shown, the electronic device 1400 includes: an image acquisition unit 1410, used to acquire an image of an application to be searched, the image including an icon of the application to be searched; a processing unit 1420, used to extract constituent elements of the image of the application to be searched, the constituent elements including graphic elements and color elements of the icon; based on the constituent elements, searching an application corresponding to the application to be searched from an application index library, wherein the application index library contains the constituent elements of images of multiple applications.

[0186] According to the technical solution in the embodiments of the present application, on the one hand, since applications are searched by image processing, applications can be efficiently searched without entering search keywords; on the other hand, since the search is performed in combination with multiple component elements of the image of the application to be searched, the accuracy of the application search results can be improved.

[0187] In a possible implementation manner, the component elements further include text elements, and the text elements include text elements in the icon and / or text elements below the icon.

[0188] In one possible implementation, the processing unit 1420 is used to: determine the element similarity between each component element of the application to be searched and the corresponding component elements of each application in the application index library, the element similarity including one or more of graphic similarity, color similarity and text similarity; determine the application corresponding to the application to be searched based on the element similarity.

[0189] In one possible implementation, the processing unit 1420 is used to include: determining multiple application sets corresponding to the application to be searched based on the element similarity, each application in the application set is at least one application whose element similarity is greater than a predetermined threshold; determining the union of each of the application sets to obtain multiple candidate applications; determining the program confidence of each candidate application based on the element similarity, the program confidence indicating the probability that the candidate application is similar to the application to be searched; and determining the application corresponding to the application to be searched based on the size of the program confidence.

[0190] In one possible implementation, the application index library also includes: the weight of the corresponding element of the candidate application, and the processing unit 1420 is used to: determine the weight of each component element of the application to be searched and the weight of the corresponding element of the candidate application; determine the element confidence of the corresponding element of the candidate application based on the weight of each component element of the application to be searched, the weight of the corresponding element of the candidate application and the corresponding element similarity, the element confidence indicates the probability that the corresponding element belongs to the element of the application to be searched; based on the element confidence, determine the program confidence of each candidate application.

[0191] In one possible implementation, the processing unit 1420 is used to: input the weights of the constituent elements of the application to be searched, the weights of the corresponding constituent elements of the candidate application, and the corresponding element similarities into an element scoring model, which is a neural network model; and determine the element confidence of the corresponding element of the candidate application through the element scoring model.

[0192] In one possible implementation, based on the element confidence, the processing unit 1420 is used to: perform a weighted sum operation on the element confidence of each component element under each category of component elements of the candidate application to determine the category element confidence of the candidate application, wherein the category element confidence indicates the probability that a category of component elements of the candidate application all belong to the application to be searched; and determine the program confidence of the candidate application based on the category element confidence.

[0193] In a possible implementation, the processing unit 1420 is used to: input the category element confidence of each category component element into a category scoring model, where the category scoring model is a neural network model; and determine the program confidence of each candidate application through the category scoring model.

[0194] In a possible implementation, the processing unit 1420 is configured to: extract graphic elements of the image of the application to be searched by using a semantic segmentation model; and extract color elements of the image of the application to be searched by using a color quantization model.

[0195] In a possible implementation, the component elements include text elements, and the processing unit 1420 is used to extract the text elements of the image of the application to be searched through a character recognition model.

[0196] In a possible implementation, the application index library further includes: element feature indexes and element features corresponding to various component elements of the application.

[0197] The electronic device 1400 provided in this embodiment is used to execute the above-mentioned application search method, and thus can achieve the same effect as the above-mentioned implementation method.

[0198] It is understandable that, in order to realize the above functions, the electronic device includes hardware and / or software modules corresponding to the execution of each function. In combination with the algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of the present application.

[0199] This embodiment also provides a computer storage medium, in which computer instructions are stored. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the application search method in the above-mentioned embodiment.

[0200] This embodiment also provides a computer program product. When the computer program product runs on a computer, it enables the computer to execute the above-mentioned related steps to implement the application search method in the above-mentioned embodiment.

[0201] In addition, an embodiment of the present application also provides a device, which may specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer execution instructions, and when the device is running, the processor may execute the computer execution instructions stored in the memory so that the chip executes the application search method in the above-mentioned method embodiments.

[0202] Among them, the electronic device, computer storage medium, computer program product or chip provided in this embodiment is used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above and will not be repeated here.

[0203] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0204] In the several embodiments provided in the present application, it should be understood that the disclosed devices, equipment and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0205] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0206] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0207] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0208] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An application search method, characterized in that: include: Acquire an image of an application to be searched, wherein the image includes an icon of the application to be searched; Extracting constituent elements of the image of the application to be searched, wherein the constituent elements include graphic elements and color elements of the icon; searching an application corresponding to the application to be searched from an application index library based on the component elements, wherein the application index library contains the component elements of images of multiple applications; The component elements further include text elements, and the text elements include text elements in the icon and / or text elements below the icon; The step of searching an application corresponding to the application to be searched from an application index library based on the component elements includes: Determine the element similarity between each component element of the application to be searched and the corresponding component element of each application in the application index library, wherein the element similarity includes one or more of graphic similarity, color similarity and text similarity; Determining an application corresponding to the application to be searched based on the element similarity; The step of determining the application corresponding to the application to be searched based on the element similarity includes: Determine a plurality of application sets corresponding to the application to be searched based on the element similarity, each application in the application set being an application having at least one element similarity greater than a predetermined threshold; Determine a union of the application sets to obtain a plurality of candidate applications; Determining a program confidence of each candidate application based on the element similarity, wherein the program confidence represents a probability that the candidate application is similar to the application to be searched; An application corresponding to the application to be searched is determined based on the magnitude of the program confidence.

2. The method according to claim 1, characterized in that The application index library further includes: weights of corresponding elements of the candidate applications, and determining the program confidence between each candidate application and the application to be searched based on the element similarity includes: Determining the weight of each component element of the application to be searched and the weight of the corresponding element of the candidate application; Determine the element confidence of the corresponding element of the candidate application based on the weight of each component element of the application to be searched, the weight of the corresponding element of the candidate application and the corresponding element similarity, wherein the element confidence indicates the probability that the corresponding element belongs to the element of the application to be searched; Based on the element confidences, a program confidence of each candidate application is determined.

3. The method according to claim 2, characterized in that The determining the element confidence of the corresponding element of the candidate application comprises: Inputting the weights of the constituent elements of the application to be searched, the weights of the corresponding constituent elements of the candidate application, and the corresponding element similarities into an element scoring model, wherein the element scoring model is a neural network model; The element confidence of the corresponding element of the candidate application is determined by the element scoring model.

4. The method according to claim 2, characterized in that: Determining the program confidence of each candidate application based on the element confidence includes: Performing a weighted sum operation on the element confidences of each component element under each category of component elements of the candidate application to determine the category element confidence of the candidate application, wherein the category element confidence indicates the probability that a category of component elements of the candidate application all belong to the application to be searched; Based on the category element confidence, a program confidence of the candidate application is determined.

5. The method according to claim 4, characterized in that The determining, based on the category element confidence, the category element confidence of the candidate application comprises: Inputting the category element confidence of each category element into a category scoring model, wherein the category scoring model is a neural network model; The program confidence of each candidate application is determined through the category scoring model.

6. The method according to any one of claims 1 to 5, characterized in that The step of extracting the constituent elements of the image of the application to be searched comprises: Extracting graphic elements of the image of the application to be searched by using a semantic segmentation model; The color elements of the image of the application to be searched are extracted through a color quantization model.

7. The method according to claim 6, characterized in that The constituent elements include text elements, and the constituent elements of the image of the application to be searched further include: The text elements of the image of the application to be searched are extracted through a character recognition model.

8. The method according to claim 6, characterized in that The application index library also includes: element feature indexes and element features corresponding to various component elements of the application.

9. An electronic device, characterized in that: The electronic device comprises: An image acquisition unit, used to acquire an image of the application to be searched, wherein the image includes an icon of the application to be searched; a processing unit, configured to extract constituent elements of the image of the application to be searched, wherein the constituent elements include graphic elements and color elements of the icon; and search an application corresponding to the application to be searched from an application index library based on the constituent elements, wherein the application index library contains the constituent elements of images of multiple applications; The component elements further include text elements, and the text elements include text elements in the icon and / or text elements below the icon; The processing unit is used for: Determine the element similarity between each component element of the application to be searched and the corresponding component element of each application in the application index library, wherein the element similarity includes one or more of graphic similarity, color similarity and text similarity; Determining an application corresponding to the application to be searched based on the element similarity; The processing unit is used for: Determine a plurality of application sets corresponding to the application to be searched based on the element similarity, each application in the application set being an application having at least one element similarity greater than a predetermined threshold; Determine a union of the application sets to obtain a plurality of candidate applications; Determining a program confidence of each candidate application based on the element similarity, wherein the program confidence represents a probability that the candidate application is similar to the application to be searched; An application corresponding to the application to be searched is determined based on the magnitude of the program confidence.

10. The device according to claim 9, characterized in that The application index library further includes: weights of corresponding elements of the candidate applications, and the processing unit is used to: Determining the weight of each component element of the application to be searched and the weight of the corresponding element of the candidate application; Determine the element confidence of the corresponding element of the candidate application based on the weight of each component element of the application to be searched, the weight of the corresponding element of the candidate application and the corresponding element similarity, wherein the element confidence indicates the probability that the corresponding element belongs to the element of the application to be searched; Based on the element confidences, a program confidence of each candidate application is determined.

11. The device according to claim 10, characterized in that The processing unit is used for: Inputting the weights of the constituent elements of the application to be searched, the weights of the corresponding constituent elements of the candidate application, and the corresponding element similarities into an element scoring model, wherein the element scoring model is a neural network model; The element confidence of the corresponding element of the candidate application is determined by the element scoring model.

12. The device according to claim 10, characterized in that Based on the element confidence, the processing unit is used for: Performing a weighted sum operation on the element confidences of each component element under each category of component elements of the candidate application to determine the category element confidence of the candidate application, wherein the category element confidence indicates the probability that a category of component elements of the candidate application all belong to the application to be searched; Based on the category element confidence, a program confidence of the candidate application is determined.

13. The device according to claim 12, characterized in that The processing unit is used for: Inputting the category element confidence of each category element into a category scoring model, wherein the category scoring model is a neural network model; The program confidence of each candidate application is determined through the category scoring model.

14. The device according to any one of claims 9 to 13, characterized in that The processing unit is used for: Extracting graphic elements of the image of the application to be searched by using a semantic segmentation model; The color elements of the image of the application to be searched are extracted through a color quantization model.

15. The device according to claim 14, characterized in that The component elements include text elements, and the processing unit is used for: The text elements of the image of the application to be searched are extracted through a character recognition model.

16. The device according to claim 14, characterized in that The application index library also includes: element feature indexes and element features corresponding to various component elements of the application.

17. A chip device, comprising: A communication interface and a processor, wherein the communication interface and the processor communicate with each other via an internal connection path, and the processor is used to implement the method according to any one of claims 1 to 8.

18. A computer-readable storage medium for storing a computer program, characterized in that: The computer program comprises instructions for implementing the method of any one of claims 1 to 8 above.

19. A computer program product, comprising instructions, characterized in that: When the instructions are executed on a computer, the computer is enabled to implement the method according to any one of claims 1 to 8.

Images