A method, device and electronic device for processing image information

Abstract

The present invention discloses a method, device and electronic device for processing image information, the method comprising: obtaining candidate images by searching for query words, obtaining a semantic vector between the query word and the image text domain of the candidate image and a site vector of the website to which the candidate image belongs, the site vector being used to represent the theme information of the website; performing semantic analysis on the query word vector of the query word and the text vector to obtain a semantic vector between the query word and the image text domain; obtaining text semantic features between the query word and the image text domain based on the site vector and the semantic vector. By adding the site vector to the calculation of the text semantic features, the semantic bias of the obtained text semantic features is consistent with the site theme, thereby making the semantic characterization more accurate, solving the technical problem of poor accuracy of the text semantic features in the semantic characterization in the prior art, and improving the accuracy of the text semantic features.

Description

Technical Field

[0001] The present invention relates to the field of search technology, and in particular to a method, device and electronic device for processing picture information. Background Art

[0002] In practical applications, search technology usually includes web page, image, map search, etc. As customer needs continue to change, the demand for image search is increasing, and the requirements for image search quality are also getting higher and higher.

[0003] In image search, the ranking of images often requires the integration of multiple dimensions of features such as text semantic features, site quality, and image aesthetic quality. Among them, text semantic features are a very important feature, which describes the semantic similarity between the user's query terms and the image text domain. Existing text semantic features are obtained by calculating the similarity between the query terms and the image text domain vectors, but in actual application, it is found that the text semantic features obtained by this method have poor accuracy in semantic description. Summary of the invention

[0004] The embodiments of the present invention provide a method, device and electronic device for processing image information, which are used to solve the technical problem of poor accuracy in semantic characterization of text semantic features in the prior art and improve the accuracy of text semantic features.

[0005] An embodiment of the present invention provides a method for processing image information, including:

[0006] Searching for a query word to obtain a candidate image, obtaining a semantic vector between the query word and the image text domain of the candidate image and a site vector of the website to which the candidate image belongs, wherein the site vector is used to represent the theme of the website;

[0007] Based on the site vector and the semantic vector, a text semantic feature between the query word and the image text domain is obtained.

[0008] Optionally, obtaining a semantic vector between the query word and the picture text field of the candidate picture includes:

[0009] Obtaining a query word vector of the query word and a text vector of the image text field;

[0010] Perform semantic analysis on the query word vector and the text vector to obtain the semantic vector.

[0011] Optionally, the method for obtaining the site vector includes:

[0012] Crawl all image text fields under each site and obtain text vectors of all image text fields;

[0013] The text vectors of all the image text fields are clustered to obtain the site vector of each site.

[0014] Optionally, clustering the text vectors of all the image text fields to obtain the site vector of each site further includes:

[0015] Crawl the article titles under each site and obtain the title vectors of the article titles;

[0016] The text vectors and title vectors of all the picture text fields under each site are clustered to obtain the site vector.

[0017] Optionally, obtaining the text semantic feature between the query term and the image text domain based on the site vector and the semantic vector includes:

[0018] Fusing the query word vector of the query word with the site vector to obtain a fused vector;

[0019] Based on the fusion vector and the semantic vector, a text semantic feature between the query word and the image text domain is obtained.

[0020] Optionally, fusing the query word vector of the query word with the site vector to obtain a fused vector includes:

[0021] Concatenate the query word vector with the site vector to obtain a concatenated vector;

[0022] A nonlinear transformation is performed on the splicing vector to obtain the fusion vector.

[0023] Optionally, obtaining the text semantic feature between the query term and the image text domain based on the fusion vector and the semantic vector includes:

[0024] The fusion vector and the semantic vector are input into a multi-layer perception network to perform similarity calculation to obtain text semantic features between the query word and the image text domain.

[0025] Optionally, the method further includes:

[0026] Sorting all candidate images corresponding to the query word based on the text semantic features to obtain a sorted image sequence;

[0027] Output image search results based on the sorted image sequence.

[0028] The embodiment of the present invention further provides a device for processing image information, including:

[0029] An acquisition unit, configured to obtain candidate images by searching for a query word, and to acquire a semantic vector between the query word and the image text domain of the candidate image and a site vector of the website to which the candidate image belongs, wherein the site vector is used to represent the theme of the website;

[0030] A calculation unit is used to obtain text semantic features between the query term and the image text domain based on the site vector and the semantic vector.

[0031] Optionally, the acquisition unit is further used for:

[0032] Obtaining a query word vector of the query word and a text vector of the image text field;

[0033] Perform semantic analysis on the query word vector and the text vector to obtain the semantic vector.

[0034] Optionally, the device further includes a generating unit, wherein the generating unit is configured to:

[0035] Crawl all image text fields under each site and obtain text vectors of all image text fields;

[0036] The text vectors of all the image text fields are clustered to obtain the site vector of each site.

[0037] Optionally, the generating unit is further used for:

[0038] Crawl the article titles under each site and obtain the title vectors of the article titles;

[0039] The text vectors and title vectors of all the picture text fields under each site are clustered to obtain the site vector.

[0040] Optionally, the computing unit is further used for:

[0041] Fusing the query word vector of the query word with the site vector to obtain a fused vector;

[0042] Based on the fusion vector and the semantic vector, a text semantic feature between the query word and the image text domain is obtained.

[0043] Optionally, the computing unit is further used for:

[0044] Concatenate the query word vector with the site vector to obtain a concatenated vector;

[0045] A nonlinear transformation is performed on the splicing vector to obtain the fusion vector.

[0046] Optionally, the computing unit is further used for:

[0047] The fusion vector and the semantic vector are input into a multi-layer perception network to perform similarity calculation to obtain text semantic features between the query word and the image text domain.

[0048] Optionally, the device further comprises a sorting unit, configured to:

[0049] Sorting all candidate images corresponding to the query word based on the text semantic features to obtain a sorted image sequence;

[0050] Output image search results based on the sorted image sequence.

[0051] The above one or more technical solutions in the embodiments of the present invention have at least the following technical effects:

[0052] The embodiment of the present invention provides a method for processing image information, which obtains candidate images by searching for query words, obtains semantic vectors between the query words and the image text domains of the candidate images and the site vectors of the websites to which the candidate images belong, and the point vectors are used to characterize the theme of the website; based on the site vectors and the semantic vectors, text semantic features between the query words and the image text domains are obtained, that is, by adding the site vectors to the calculation of the text semantic features, the semantic bias of the obtained text semantic features is consistent with the site theme, thereby making the semantic characterization more accurate, thereby solving the technical problem of poor accuracy of text semantic features in semantic characterization in the prior art and improving the accuracy of text semantic features. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 A schematic diagram of a process flow of a method for processing image information provided by an embodiment of the present invention;

[0054] Figure 2 A block diagram of a device for processing picture information provided by an embodiment of the present invention;

[0055] Figure 3 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention;

[0056] Figure 4 A schematic diagram of the structure of a server provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0057] In the technical solution provided in the embodiment of the present invention, a method for processing image information is provided, which improves the accuracy of text semantic features in semantic characterization by adding a site vector representing a website theme into the calculation of text semantic features, thereby solving the technical problem of poor accuracy of text semantic features in semantic characterization in the prior art.

[0058] The main implementation principles, specific implementation methods and corresponding beneficial effects of the technical solutions of the embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0059] Example

[0060] Please refer to Figure 1 , an embodiment of the present invention provides a method for processing image information, the method comprising:

[0061] S11, searching for a candidate image for a query word, obtaining a semantic vector between the query word and the image text domain of the candidate image and a site vector of the website to which the candidate image belongs, wherein the site vector is used to represent the theme of the website;

[0062] S13. Obtaining text semantic features between the query term and the image text domain based on the site vector and the semantic vector.

[0063] In the specific implementation process, multiple candidate images will be obtained by searching for images through query words. In order to sort the multiple candidate images, it is necessary to further obtain the text semantic features between the query word query and the image text domain doc of the candidate images. This embodiment additionally considers the influence of the website on the semantic bias and text quality of the image text domain doc. When obtaining the text semantic features, S11 is first executed to obtain the site vector of the website and the text domain vector of the image text domain. S11 can search and obtain the pre-stored site vector and text domain vector corresponding to the candidate image from the image database.

[0064] The text field vector can be obtained by vector conversion of the image text field. The image text field can be an image title, an image title and image surrounding text, or a collection of an image title, image surrounding text and other image-related text such as text within the image.

[0065] The site vector can be obtained by the following method: for each website, crawl all the image text domains under the website and obtain the text domain vectors of all the image text domains; cluster the text domain vectors of all the image text domains to obtain the site vector of the website. Furthermore, the article titles and title vectors of the article titles of each website can be obtained; the text domain vectors and title vectors of all the image text domains of each website are clustered to obtain the site vector of the website. When clustering, each cluster obtained by clustering represents a topic semantics under the website, and the average of the central vectors of all clusters is taken as the site vector of the website.

[0066] For example: for the website hcxxx.com, the website contains n pictures, and n picture text domains corresponding to the n pictures are obtained, as well as the article titles of the website; each picture text domain and article title are respectively converted into vectors through a general language model to obtain n text domain vectors and title vectors; these n text domain vectors and title vectors are clustered, assuming that m clusters are obtained by clustering, each cluster corresponds to a central vector, and the m central vectors are averaged to obtain the site vector. Different websites have different themes. For example, some websites tend to be entertainment gossip, some tend to be industrial products, and some tend to be clothing products. This embodiment uses site vectors to characterize different themes of different websites. In the specific implementation process, in order to avoid wasting computing resources, the websites corresponding to all the picture text domains doc in the database can be counted first to obtain a certain number of websites with the highest frequency of occurrence, such as the 100,000 websites with the most occurrences, and their site vectors are calculated for these websites.

[0067] At the same time, before or after obtaining the site vector, S11 also obtains the semantic vector between the query word and the image text domain of the candidate image. Among them, the semantic vector is a multi-dimensional vector that represents the similarity between the query word and the image text domain. Specifically, the query word vector of the query word and the text vector of the image text domain can be obtained first, and then the query word vector and the text vector are semantically analyzed to obtain the semantic vector. For example, the query word vector and the text domain vector are input into a text semantic model, such as a multi-layer long short-term memory network LSTM, a one-dimensional convolutional neural network or a BERT network, and the semantic vector is output by performing semantic analysis through the text semantic model.

[0068] After obtaining the semantic vector and the site vector, execute S13 to obtain the text semantic features between the query word and the image text domain based on the semantic vector and the site vector. Specifically, the query word vector and the site vector can be fused to obtain a fused vector; based on the fused vector and the semantic vector, the text semantic features between the query word and the image text domain are obtained. The fusion of the query word vector and the site vector can fuse the query word information into the site vector to obtain a fused vector that fuses the query word information.

[0069] When the query word vector is fused with the site vector, the query word vector and the site vector can be concatenated to obtain a concatenated vector; the concatenated vector can be nonlinearly changed to obtain a fused vector. The nonlinear change of the concatenated vector can be achieved through a multi-layer perceptron network. For example: for the query word vector q and the site vector s, the concatenated vector (q, s) is concatenated, and then (q, s) is input into the multi-layer perceptron network to obtain the vector s' output by the multi-layer perceptron network as the fused vector. When obtaining semantic text features based on the fused vector and the semantic vector, the fused vector and the semantic vector can also be nonlinearly changed to obtain a semantic score as a text semantic feature. For example, the fused vector and the semantic vector are input into the multi-layer perceptron network for similarity calculation to obtain the text semantic features between the query word and the image text domain.

[0070] In the specific implementation process, after obtaining the text semantic vector between the query word and the image text domain in S13, all candidate images corresponding to the query word can also be sorted based on the text semantic features to obtain a sorted image sequence; and the image search results are output according to the sorted image sequence. Alternatively, multiple features such as the text semantic features between the query word and the image text domain, the site quality, and the image aesthetic quality are input into the image sorting model, and the sorted image sequence is output through the image sorting model, and the image search results are output according to the image sequence.

[0071] In the above embodiment, by obtaining the text semantic features, i.e., the semantic score, between the query word query and the image text domain doc based on the site vector and the semantic vector, the site information of the image text domain doc is introduced into the calculation of the semantic score. In this way, the quality and subject differences of different sites and the matching degree between the query and doc sites can be taken into account, so that the relevance between the query and doc can be evaluated more accurately, and then a more accurate image ranking result can be obtained, thereby improving the quality of image search.

[0072] A method for processing image information is provided in accordance with the above embodiment. The present invention also provides a device for processing image information. Figure 2 , the device comprises:

[0073] An acquisition unit 21 is used to search for a query word to obtain a candidate image, obtain a semantic vector between the query word and the image text domain of the candidate image and a site vector of the website to which the candidate image belongs, wherein the site vector is used to represent the theme of the website;

[0074] The calculation unit 22 is used to obtain the text semantic feature between the query word and the image text domain based on the site vector and the semantic vector.

[0075] As an optional implementation, the acquisition unit 21 is further used to: obtain a query word vector of the query word and a text vector of the image text field; and perform semantic analysis on the query word vector and the text vector to obtain the semantic vector.

[0076] As an optional implementation, the device further includes a generation unit 23, the generation unit 23 is used to: crawl all image text fields under each site, and obtain text vectors of all image text fields; cluster the text vectors of all image text fields to obtain the site vector of each site. Further, the generation unit 23 can also be used to crawl article titles under each site, and obtain title vectors of the article titles; cluster the text vectors and title vectors of all image text fields under each site to obtain the site vector.

[0077] As an optional implementation, the calculation unit 22 is further used to: fuse the query word vector of the query word with the site vector to obtain a fusion vector; and obtain text semantic features between the query word and the image text domain based on the fusion vector and the semantic vector.

[0078] When obtaining the fusion vector, the query word vector and the site vector may be spliced ​​to obtain a spliced ​​vector; the spliced ​​vector may be subjected to nonlinear transformation to obtain the fusion vector. Further, the fusion vector and the semantic vector are input into a multi-layer perception network for similarity calculation to obtain the text semantic features between the query word and the image text domain.

[0079] As an optional implementation, the device further includes a sorting unit 24 for sorting all candidate images corresponding to the query term based on the text semantic features to obtain a sorted image sequence; and outputting image search results based on the sorted image sequence.

[0080] Regarding the device in the above embodiment, the specific manner in which each unit performs the operation has been described in detail in the embodiment of the method, and will not be elaborated here.

[0081] Figure 3 1 is a block diagram of an electronic device 300 for implementing a method for processing picture information according to an exemplary embodiment. For example, the electronic device 300 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0082] Reference Figure 3, the electronic device 300 may include one or more of the following components: a processing component 302 , a memory 304 , a power component 306 , a multimedia component 308 , an audio component 310 , an input / display (I / O) interface 312 , a sensor component 314 , and a communication component 316 .

[0083] The processing component 302 generally controls the overall operation of the electronic device 300, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 302 may include one or more processors 320 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 302 may include one or more modules to facilitate the interaction between the processing component 302 and other components. For example, the processing component 302 may include a multimedia module to facilitate the interaction between the multimedia component 308 and the processing component 302.

[0084] The memory 304 is configured to store various types of data to support operations on the device 300. Examples of such data include instructions for any application or method operating on the electronic device 300, contact data, phone book data, messages, pictures, videos, etc. The memory 304 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0085] The power supply component 306 provides power to the various components of the electronic device 300. The power supply component 306 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 300.

[0086] The multimedia component 308 includes a screen that provides a display interface between the electronic device 300 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 308 includes a front camera and / or a rear camera. When the device 300 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0087] The audio component 310 is configured to present and / or input audio signals. For example, the audio component 310 includes a microphone (MIC), and when the electronic device 300 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 304 or sent via the communication component 316. In some embodiments, the audio component 310 also includes a speaker for presenting audio signals.

[0088] I / O interface 312 provides an interface between processing component 302 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0089] The sensor assembly 314 includes one or more sensors for providing various aspects of status assessment for the electronic device 300. For example, the sensor assembly 314 can detect the open / closed state of the device 300, the relative positioning of components, such as the display and keypad of the electronic device 300, and the sensor assembly 314 can also detect the position change of the electronic device 300 or a component of the electronic device 300, the presence or absence of user contact with the electronic device 300, the orientation or acceleration / deceleration of the electronic device 300, and the temperature change of the electronic device 300. The sensor assembly 314 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 314 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 314 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0090] The communication component 316 is configured to facilitate wired or wireless communication between the electronic device 300 and other devices. The electronic device 300 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 316 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 316 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0091] In an exemplary embodiment, the electronic device 300 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0092] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 304 including instructions, and the instructions can be executed by a processor 320 of an electronic device 300 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0093] A non-temporary computer-readable storage medium, when the instructions in the storage medium are executed by the processor of a mobile terminal, enables the mobile terminal to execute a method for processing image information, the method comprising: obtaining candidate images by searching for query words, obtaining a semantic vector between the query words and the image text domain of the candidate images and a site vector of the website to which the candidate images belong, the site vector being used to characterize the theme of the website; and obtaining text semantic features between the query words and the image text domain based on the site vector and the semantic vector.

[0094] Figure 44 is a schematic diagram of the structure of the server in the embodiment of the present invention. The server 400 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 422 (for example, one or more processors) and memory 432, and one or more storage media 430 (for example, one or more mass storage devices) storing application programs 442 or data 444. Among them, the memory 432 and the storage medium 430 can be short-term storage or permanent storage. The program stored in the storage medium 430 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the server. Furthermore, the central processing unit 422 can be configured to communicate with the storage medium 430 to execute a series of instruction operations in the storage medium 430 on the server 400.

[0095] The server 400 may also include one or more power supplies 426, one or more wired or wireless network interfaces 450, one or more input presentation interfaces 458, one or more keyboards 456, and / or, one or more operating systems 441, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0096] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed in this disclosure. The specification and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.

[0097] It should be understood that the present invention is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims. The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention shall be included in the scope of protection of the present invention.

Claims

1. A method for processing image information, characterized in that: include: Searching for a query word to obtain a candidate image, obtaining a semantic vector between the query word and the image text domain of the candidate image and a site vector of the website to which the candidate image belongs, wherein the site vector is used to represent the theme of the website; The method for obtaining the site vector includes: Crawl all image text fields under each site and obtain text vectors of all image text fields; Clustering the text vectors of all the image text fields to obtain the site vector of each site; Based on the site vector and the semantic vector, a text semantic feature between the query word and the image text domain is obtained.

2. The method according to claim 1, characterized in that The obtaining of the semantic vector between the query word and the picture text field of the candidate picture includes: Obtaining a query word vector of the query word and a text vector of the image text field; Perform semantic analysis on the query word vector and the text vector to obtain the semantic vector.

3. The method according to claim 1, characterized in that The clustering of the text vectors of all the image text fields to obtain the site vector of each site further includes: Crawl the article titles under each site and obtain the title vectors of the article titles; The text vectors and title vectors of all the picture text fields under each site are clustered to obtain the site vector.

4. The method according to claim 1 or 2, characterized in that: The obtaining, based on the site vector and the semantic vector, a text semantic feature between the query word and the image text domain includes: Fusing the query word vector of the query word with the site vector to obtain a fused vector; Based on the fusion vector and the semantic vector, a text semantic feature between the query word and the image text domain is obtained.

5. The method according to claim 4, characterized in that The step of fusing the query word vector of the query word with the site vector to obtain a fused vector includes: Concatenate the query word vector with the site vector to obtain a concatenated vector; A nonlinear transformation is performed on the splicing vector to obtain the fusion vector.

6. The method according to claim 4, characterized in that The obtaining, based on the fusion vector and the semantic vector, a text semantic feature between the query word and the image text domain includes: The fusion vector and the semantic vector are input into a multi-layer perception network to perform similarity calculation to obtain text semantic features between the query word and the image text domain.

7. A device for processing image information, characterized in that: include: An acquisition unit, configured to obtain candidate images by searching for a query word, and to acquire a semantic vector between the query word and the image text domain of the candidate image and a site vector of the website to which the candidate image belongs, wherein the site vector is used to represent the theme of the website; A generating unit, used for crawling all image text fields under each site and obtaining text vectors of all image text fields; Clustering the text vectors of all the image text fields to obtain the site vector of each site; A calculation unit is used to obtain text semantic features between the query term and the image text domain based on the site vector and the semantic vector.

8. An electronic device, characterized in that: It includes a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors to execute the operation instructions corresponding to the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps corresponding to the method according to any one of claims 1 to 6 are implemented.

Images