Data description method and device, computer readable medium and electronic device
By extracting and matching the feature information of the input data, cross-modal characterization description is realized, and the problems of low matching accuracy and difficult deployment of text retrieval images and image generation text tasks in the prior art are solved, and the accuracy and efficiency are improved. Reduced costs.
Patent Information
- Application Number
- CN202110793870.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-07-14
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2041-07-14
AI Technical Summary
In the prior art, the task matching accuracy of text retrieval images and image generation text is low, and it is costly and difficult when deploying a mobile terminal, mainly due to the large amount of computing and model complexity.
By extracting the first feature information of the input data and the second feature information of the target data type, matching based on cross-modal characterization description data, avoiding information loss during direct conversion, reducing the amount of calculation, and narrowing the matching range by actively selecting the data type .
It improves matching accuracy, reduces computing volume and deployment cost, simplifies deployment difficulty, and improves data matching conversion efficiency.
Smart Images

Figure CN113536009B_ABST
Abstract
Description
Background Art
[0002] With the continuous improvement of people's living standards, mobile terminals such as smart phones and iPads are becoming more and more important in people's lives. Text retrieval image technology and image generation text technology are the two most common image-text multimodal tasks.
[0003] At present, the tasks of text retrieval from images and image generation from text in related technical solutions not only have low matching accuracy, but also require a large amount of computing power, resulting in high deployment costs and difficulty when deployed on the terminal side, especially the mobile terminal side. Summary of the invention
[0004] The purpose of the present disclosure is to provide a data description method, a data description device, a computer-readable medium and an electronic device, thereby overcoming, at least to a certain extent, the low matching accuracy between the task of text retrieval from images and the task of generating text from images, as well as the high deployment cost and difficulty when deployed to a terminal side, especially a mobile terminal side.
[0005] According to a first aspect of the present disclosure, a data description method is provided, comprising:
[0006] Get input data;
[0007] Determining first characteristic information of the input data and determining at least one target data type for describing the input data;
[0008] Acquire description data corresponding to the target data type and second characteristic information associated with the description data;
[0009] The target description data is matched from the description data according to the first feature information and the second feature information, so as to describe the input data in a cross-modal representation based on the target description data.
[0010] According to a second aspect of the present disclosure, there is provided a data description device, comprising:
[0011] An input data acquisition module, used for acquiring input data;
[0012] A first feature information extraction module, used to determine first feature information of the input data and to determine at least one target data type used to describe the input data;
[0013] A second characteristic information acquisition module, used to acquire description data corresponding to the target data type and second characteristic information associated with the description data;
[0014] A target description data screening module is used to match target description data from the description data according to the first feature information and the second feature information, so as to describe the input data based on the cross-modal representation of the target description data.
[0015] According to a third aspect of the present disclosure, a computer-readable medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the above method is implemented.
[0016] According to a fourth aspect of the present disclosure, there is provided an electronic device, characterized in that it includes:
[0017] Processor; and
[0018] The memory is used to store one or more programs. When the one or more programs are executed by one or more processors, the one or more processors implement the above method.
[0019] A data description method provided by an embodiment of the present disclosure first determines the first feature information of the input data and determines at least one target data type used to describe the input data, then obtains the second feature information of the description data corresponding to the target data type, and finally determines the target description data from the description data corresponding to the target data type according to the first feature information and the second feature information, and then can describe the input data through the cross-modal representation of the target description data. On the one hand, the first feature information of the input data and the second feature information of the description data are extracted, and the target description data is screened and matched through the first feature information and the second feature information, thereby avoiding the information loss generated when the input data is directly converted into the description data, and improving the accuracy of the matched target description data; on the other hand, the conversion between the semantically complex input data and the description data is converted into the matching between the semantically simple first feature information and the second feature information, effectively reducing the amount of calculation, reducing the deployment cost and the deployment difficulty; on the other hand, by actively selecting the data type used to describe the input data and narrowing the scope of data screening and matching, the amount of calculation can be further reduced and the data matching conversion efficiency can be improved.
[0020] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:
[0022] Figure 1 A schematic diagram showing an exemplary system architecture to which embodiments of the present disclosure may be applied;
[0023] Figure 2 A schematic diagram showing an electronic device to which the embodiments of the present disclosure can be applied;
[0024] Figure 3 A flowchart schematically illustrates a data description method in an exemplary embodiment of the present disclosure;
[0025] Figure 4 A flowchart of extracting first feature information corresponding to input data in an exemplary embodiment of the present disclosure is schematically shown;
[0026] Figure 5 A flowchart for implementing text data search for image data in an album application in an exemplary embodiment of the present disclosure is schematically shown;
[0027] Figure 6 A flowchart for describing the content of image data by text data in an exemplary embodiment of the present disclosure is schematically shown;
[0028] Figure 7 A flowchart of generating target text data based on image tags in an exemplary embodiment of the present disclosure is schematically shown;
[0029] Figure 8 A flowchart for calculating the similarity between first feature information and second feature information in an exemplary embodiment of the present disclosure is schematically shown;
[0030] Fig. 9 A flowchart of matching and screening target description data in an exemplary embodiment of the present disclosure is schematically shown;
[0031] Fig.10 The following is a schematic diagram schematically showing the composition of a data description device in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION
[0032] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
[0033] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.
[0034] Figure 1 A schematic diagram of a system architecture of an exemplary application environment to which a data description method and device according to an embodiment of the present disclosure can be applied is shown.
[0035] like Figure 1 As shown, the system architecture 100 may include one or more of terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc. The terminal devices 101, 102, 103 may be various electronic devices with image processing functions, including but not limited to desktop computers, portable computers, smart phones, tablet computers, etc. It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. According to the implementation requirements, there may be any number of terminal devices, networks and servers. For example, the server 105 may be a server cluster composed of multiple servers.
[0036] The data description method provided in the embodiment of the present disclosure is generally executed by the terminal devices 101, 102, and 103, and accordingly, the data description device is generally set in the terminal devices 101, 102, and 103. However, it is easy for those skilled in the art to understand that the data description method provided in the embodiment of the present disclosure can also be executed by the server 105, and accordingly, the data description device can also be set in the server 105, which is not particularly limited in this exemplary embodiment. For example, in an exemplary embodiment, the user can upload the acquired input data to the server 105 through the terminal devices 101, 102, and 103, and after the server generates the target description data through the data description method provided in the embodiment of the present disclosure, the target description data is transmitted to the terminal devices 101, 102, and 103, etc.
[0037] The exemplary embodiment of the present disclosure provides an electronic device for implementing a data description method, which may be Figure 1The terminal device 101, 102, 103 or the server 105 in the embodiment of the present invention comprises at least a processor and a memory, the memory is used to store executable instructions of the processor, and the processor is configured to execute the data description method by executing the executable instructions.
[0038] Below Figure 2 Taking the mobile terminal 200 in FIG. 1 as an example, the structure of the electronic device is exemplarily described. It should be understood by those skilled in the art that, in addition to the components specifically used for mobile purposes, Figure 2 The structure in the figure can also be applied to fixed type devices. In other embodiments, the mobile terminal 200 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure can be implemented in hardware, software, or a combination of software and hardware. The interface connection relationship between the components is only shown schematically and does not constitute a structural limitation of the mobile terminal 200. In other embodiments, the mobile terminal 200 may also adopt the same Figure 2 Different interface connection methods, or a combination of multiple interface connection methods.
[0039] like Figure 2 As shown, the mobile terminal 200 may specifically include: a processor 210, an internal memory 221, an external memory interface 222, a Universal Serial Bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 271, a receiver 272, a microphone 273, an earphone interface 274, a sensor module 280, a display screen 290, a camera module 291, an indicator 292, a motor 293, a button 294, and a subscriber identification module (SIM) card interface 295, etc. The sensor module 280 may include a depth sensor 2801, a pressure sensor 2802, a gyroscope sensor 2803, etc.
[0040] The processor 210 may include one or more processing units, for example, the processor 210 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor and / or a neural network processor (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.
[0041] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of the mobile terminal 200 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.
[0042] The processor 210 is provided with a memory. The memory can store instructions for implementing six modular functions: detection instructions, connection instructions, information management instructions, analysis instructions, data transmission instructions, and notification instructions, and the execution is controlled by the processor 210.
[0043] The charging management module 240 is used to receive charging input from the charger. The power management module 241 is used to connect the battery 242, the charging management module 240 and the processor 210. The power management module 241 receives input from the battery 242 and / or the charging management module 240 to power the processor 210, the internal memory 221, the display screen 290, the camera module 291 and the wireless communication module 260.
[0044] The wireless communication function of the mobile terminal 200 can be implemented through antenna 1, antenna 2, mobile communication module 250, wireless communication module 260, modulation and demodulation processor and baseband processor. Among them, antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals; mobile communication module 250 can provide solutions for wireless communication including 2G / 3G / 4G / 5G applied on mobile terminal 200; modulation and demodulation processor can include modulator and demodulator; wireless communication module 260 can provide solutions for wireless communication including wireless local area network (WLAN) (such as wireless fidelity (Wi-Fi) network), Bluetooth (BT) and the like applied on mobile terminal 200. In some embodiments, antenna 1 of mobile terminal 200 is coupled with mobile communication module 250, and antenna 2 is coupled with wireless communication module 260, so that mobile terminal 200 can communicate with network and other devices through wireless communication technology.
[0045] The mobile terminal 200 implements the display function through a GPU, a display screen 290, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 290 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 210 may include one or more GPUs that execute program instructions to generate or change display information.
[0046] The mobile terminal 200 can realize the shooting function through the ISP, the camera module 291, the video codec, the GPU, the display screen 290 and the application processor, etc. Among them, the ISP is used to process the data fed back by the camera module 291; the camera module 291 is used to capture static images or videos; the digital signal processor is used to process digital signals, and can process other digital signals in addition to digital image signals; the video codec is used to compress or decompress digital videos, and the mobile terminal 200 can also support one or more video codecs.
[0047] The external memory interface 222 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the mobile terminal 200. The external memory card communicates with the processor 210 through the external memory interface 222 to implement a data storage function, such as storing music, video and other files in the external memory card.
[0048] The internal memory 221 can be used to store computer executable program codes, and the executable program codes include instructions. The internal memory 221 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the mobile terminal 200 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 221 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash memory (Universal Flash Storage, UFS), etc. The processor 210 executes various functional applications and data processing of the mobile terminal 200 by running instructions stored in the internal memory 221 and / or instructions stored in a memory provided in the processor.
[0049] The mobile terminal 200 can implement audio functions such as music playing and recording through the audio module 270 , the speaker 271 , the receiver 272 , the microphone 273 , the earphone interface 274 and the application processor.
[0050] The depth sensor 2801 is used to obtain the depth information of the scene. In some embodiments, the depth sensor can be arranged in the camera module 291.
[0051] The pressure sensor 2802 is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 2802 can be arranged on the display screen 290. There are many types of pressure sensors 2802, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc.
[0052] The gyro sensor 2803 can be used to determine the motion posture of the mobile terminal 200. In some embodiments, the angular velocity of the mobile terminal 200 around three axes (i.e., x, y, and z axes) can be determined by the gyro sensor 2803. The gyro sensor 2803 can be used for shooting anti-shake, navigation, somatosensory game scenes, etc.
[0053] In addition, sensors with other functions may be set in the sensor module 280 according to actual needs, such as an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc.
[0054] The mobile terminal 200 may also include other devices that provide auxiliary functions. For example, the key 294 includes a power key, a volume key, etc., and the user can generate key signal input related to the user settings and function control of the mobile terminal 200 through key input. For example, the indicator 292, the motor 293, the SIM card interface 295, etc.
[0055] At present, the relevant technical solutions for retrieving images through sentences are to extract the vector representation of text data and image data through two network branches respectively, and then train the text and image features through the text and image pair information, and then input the image data, and search the text library for the nearest text data to obtain the retrieval results. However, in this technical solution, the semantics contained in the text data are much more complex. The current text and image data is at most 1 picture corresponding to 5 sentences, and the total number of pictures is in the hundreds of thousands. It is difficult to obtain the semantic features of the common feature space of the two directly through cross-modality, that is, the output embedding vector has low discrimination, resulting in low retrieval accuracy.
[0056] In the related technical solution of describing images through text, the image data is encoded by CNN (Convolutional Neural Networks) encoder, and then the corresponding text data is generated by RNN (Recurrent Neural Network) / transformer decoder. However, in this technical solution, the keywords described are inaccurate and uncontrollable because the RNN decoder often generates text based on the text sequence that often appears in the training set, ignoring the semantic information on the image; at the same time, the model is large and non-parallel, and the deployment cost is high.
[0057] Based on one or more technical problems in the related technical solutions, a data description method is first provided. The data description method and data description device of the exemplary implementation mode of the present disclosure are specifically described below by taking the execution of a mobile terminal as an example.
[0058] Figure 3 The flow of a data description method in this exemplary embodiment is shown, including the following steps S310 to S340:
[0059] In step S310, input data is obtained.
[0060] In an exemplary embodiment, the input data may be data that needs to be converted or described into other data types. For example, the input data may be text data or image data. Of course, it may also be voice audio data or music audio data, etc. This exemplary embodiment does not specifically limit this.
[0061] The input data input by the user can be obtained by setting the input interface. Of course, different input data can also be provided to the user so that the user can enter the input data through selection operations. The input data can also be in a database. This example embodiment does not impose any special restrictions on the source of the input data.
[0062] In step S320, first characteristic information of the input data is determined and at least one target data type for describing the input data is determined.
[0063] In an exemplary embodiment, the first feature information refers to data extracted from the input data for characterizing the key content of the input data. For example, when the input data is text data, the first feature information may be a keyword corresponding to the text data. When the input data is image data, the first feature information may be an image label identified from the image data. The first feature information is associated with the data type of the input data.
[0064] The target data type refers to the data type corresponding to the description data used to describe the input data. For example, when the input data is text data, the selected target data type is an image type, then the description data obtained by matching is the image data corresponding to the text data. Of course, the target data type can also be a voice type, a music / accompaniment type, etc. Multiple target data types can be selected. For example, when the input data is text data, multiple data types can be provided for the user to choose. Assuming that the user selects a music / accompaniment type and an image type, the description data obtained after conversion is the music / accompaniment data and the image data corresponding to the text data. Of course, the number of times is only a schematic example and should not impose any special limitations on this example embodiment.
[0065] In step S330, description data corresponding to the target data type and second characteristic information of the description data are obtained.
[0066] In an exemplary embodiment, the second feature information refers to data extracted from the description data for characterizing the key content of the description data. For example, when the description data is text data, the second feature information may be keywords corresponding to the text data. When the description data is image data, the second feature information may be an image tag identified from the image data. The second feature information is associated with the data type of the description data.
[0067] It should be noted that the “first” and “second” in the “first characteristic information” and “second characteristic information” in this example embodiment are only used to distinguish the characteristic information between input data and description data, and have no special meaning in essence, and should not cause any special limitation to this example embodiment.
[0068] In step S340, target description data is matched from the description data according to the first feature information and the second feature information, so as to describe the input data in a cross-modal representation based on the target description data.
[0069] In an exemplary embodiment, the target description data refers to the description data that is most suitable for describing the input data and is screened and matched from the description data corresponding to the target data type. Specifically, the target description data can be one or more, and can be one data type or multiple data types. This exemplary embodiment does not make any special restrictions on this. For example, the input data can be text data, then the target description data that is screened and matched can be one or more image data for describing the text data. Of course, in the case where multiple target data types are selected, such as the image type and the voice type, the target description data can also be one or more image data for describing the text data, and one or more voice data for describing the text data. This exemplary embodiment does not make any special restrictions on this.
[0070] Next, steps S310 to S340 are described in detail.
[0071] In an exemplary embodiment, the description data may include at least text data and image data, which may be obtained by Figure 4 The steps in the implementation of the conversion description of determining text data and image data are described in detail. Figure 4 As shown, it may specifically include:
[0072] Step S410, extracting keywords corresponding to the text data, and associating the keywords with the text data as description data of the text data type;
[0073] Step S420, extracting an image tag corresponding to the image data, and associating the image tag with the image data as description data of the image data type;
[0074] Step S430, determining a matching relationship between the description data of the text data type and the description data of the image data type based on the keyword and the image tag, so as to implement mutual conversion description between the text data and the image data according to the matching relationship.
[0075] Among them, keywords refer to words that can represent the key semantic information of text data. For example, for the text data "I went to the park to exercise today", the corresponding keywords are "I", "today", "park", and "exercise". The text data is associated with the extracted keywords and used as descriptive data of the text data type to describe the input data of other data types.
[0076] Image Tagging refers to accurately identifying the visual content in an image based on deep learning technology and tagging various object objects and relevant tags of the scene. Image tags help a computer or user accurately identify and understand the image content. For example, for an image of a child flying a kite on the grass, the image tags can be "child", "grass", "kite". Associating the image tags with the image data serves as descriptive data of the image data type and is used to describe the input data of other data types.
[0077] Specifically, the similarity between keywords and image tags can be pre-computed, and based on this similarity, a matching relationship between the descriptive data of the text data type and the descriptive data of the image data type can be pre-constructed. Furthermore, based on this matching relationship, the mutual conversion description between text data and image data can be achieved. In this way, the mutual conversion description between text data and image data can be unified, reducing the complexity of the conversion model between text data and image data, enabling the mutual conversion description function between text data and image data to be deployed on a mobile terminal with a relatively small amount of computation, and reducing the deployment cost of the model.
[0078] Specifically, the text data can be segmented to extract the keywords of the text data, and the image tags of the image data can be extracted through a multi-label model.
[0079] Among them, the segmentation process refers to the process of cutting text data into multiple combinations of words. For example, the text data can be segmented by means of string matching (such as by matching a word dictionary), or by means of semantic analysis (such as based on a natural language processing model). Of course, it can also be other ways that can segment the text data. This exemplary embodiment does not make special limitations on this.
[0080] After segmenting the text data, meaningless words in the segmented data can be deleted, such as "de", "ne" in the text data. By deleting meaningless words, the number of matching keywords is reduced, and the computation amount during screening and matching is reduced.
[0081] For example, assume the text data is "I went to the park to exercise today". After word segmentation, we get "I", "today", "go", "to", "the", "park", "exercise", "today". Deleting the meaningless words "go" and "today" from this data, we obtain the keywords corresponding to the text data "I", "today", "park", "exercise", and use the keywords "I", "today", "park", "exercise" as the first feature information of the text data "I went to the park to exercise today".
[0082] The multi-label model is a multi-label classification model, which can determine one or more labels corresponding to the input data. Specifically, before training the multi-label model, a label system for identifying data such as image data can be pre-constructed. For example, the label system of image data can be mainly divided into four categories of labels: subject labels, scene labels, behavior labels, and information labels. These four categories of labels can then be further divided into more detailed labels. The more detailed the label system classification is, the richer and more accurate the labels of the generated images will be. Based on the constructed label system training, a multi-label model for identifying image data is obtained. By inputting the image data into the multi-label model, multiple image labels corresponding to the image data can be output, and the multiple image labels are used as the first feature information of the image data.
[0083] For example, for an image of a child flying a kite on the grass, the image is input into a multi-label classification model for image label extraction, and the obtained image labels may be "child", "grass", and "kite". Of course, this is only a schematic example and this example embodiment does not specifically limit this.
[0084] Compared with the encoder-decoder based target detection model, the multi-label classification model is used to identify the image labels corresponding to the image data. It has the advantages of simple model structure, simple and rich training data annotation, and high model accuracy. At the same time, the multi-label classification model solidifies the objects and scenes in the image, which is more in line with the process of human description of images.
[0085] In an exemplary embodiment, the data description method can be applied to an application scenario of searching for images in a photo album application. Figure 5 As shown, the function of searching for images in the photo album application can be implemented by following the steps below, which may include:
[0086] Step S510, obtaining text data input in the photo album application, and determining keywords of the text data;
[0087] Step S520, obtaining image tags corresponding to the album images stored in the album application, and calculating the similarity between the keyword and the image tag;
[0088] Step S530 , determining target album images from the album images according to the similarity, and sorting and displaying the target album images according to the similarity.
[0089] The album application refers to an application program for displaying and storing photo images taken by a user, and the target album image refers to an image corresponding to the text data that is screened and matched from a plurality of album images stored in the album application.
[0090] In one application scenario of this example embodiment, a search interface can be provided in an album application. The user enters text data in the search interface, and when an album image corresponding to the text data is matched, the album image is displayed to the user in the album application, thereby realizing the function of searching for images through a sentence.
[0091] Specifically, first, the text data input in the album application can be obtained, and the keywords contained in the text data can be determined by performing word segmentation on the text data. Then, the image tags pre-extracted from the album images stored in the album application can be obtained, the word vectors corresponding to the keywords and the word vectors corresponding to the image tags can be determined, and the Euclidean distance between the word vectors corresponding to the keywords and the word vectors corresponding to the image tags can be calculated to obtain the similarity between the keywords and the image tags. Then, the album images associated with the image tags whose similarity is greater than or equal to the similarity threshold can be used as the target album images corresponding to the text data.
[0092] By matching keywords with image tags, target album images that match text data can be quickly screened from album images. The matching efficiency is high and the accuracy of the matching results is high. At the same time, the entire solution has a small amount of computation and can be deployed on mobile terminals with low deployment cost and difficulty. The function of quickly searching for album images with corresponding content in the album through the input text data is realized.
[0093] Furthermore, the target album images obtained by screening and matching can be sorted according to the similarity between the keyword and the image tag. For example, the target album images can be sorted in descending order of similarity. Of course, the target album images can also be sorted in ascending order of similarity. This can be customized according to the user's settings, and this exemplary embodiment does not specifically limit this. By sorting the target album images, it is convenient for users to quickly find album images that meet their expectations.
[0094] In an exemplary embodiment, the data description method can be applied to an application scenario in which the content contained in the image data is described by text data, referring to Figure 6 As shown, the function of describing the content contained in the image data by text data can be realized by the following steps, which may specifically include:
[0095] Step S610, acquiring image data to be described, and determining an image tag of the image data;
[0096] Step S620, obtaining keywords of text data stored in a database, calculating similarities between the image tags and the keywords, and determining target text data from the text data according to the similarities; or
[0097] Step S630: generating target text data according to the image tag.
[0098] Among them, the image data to be described can be image data input through the provided input interface. Of course, the image data to be described can also be album images stored in the album application. The user can trigger the image description function provided in the album application (for example, long pressing the album image or double-clicking the album image to perform other triggering operations) to use the album image as the image data to be described.
[0099] In an application scenario of this example embodiment, the image data to be described can be input through a preset input interface or in an album application, and then the image tags contained in the image data to be described can be extracted. At the same time, the keywords of the text data stored in the database can be obtained, the word vectors corresponding to the image tags and the word vectors corresponding to the keywords can be determined, and the Euclidean distance between the word vectors corresponding to the keywords and the word vectors corresponding to the image tags is calculated to obtain the similarity between the keywords and the image tags, and then the text data associated with the keywords whose similarity is greater than or equal to the similarity threshold can be used as the target text data corresponding to the image data to be described.
[0100] In this example embodiment, in addition to filtering the corresponding target text data from the database according to the similarity between the keyword and the image tag, the image tag can also be assembled to realize the generation of the target text data. Figure 7 As shown, it may specifically include:
[0101] Step S710, determining the arrangement order and connecting words between the image tags according to a preset text grammar model;
[0102] Step S720: generating target text data for describing the image data through the arrangement order, the connecting words, and the image tags.
[0103] Among them, the text grammar model refers to a natural language processing model that can sort multiple words according to certain semantic and syntactic features. Of course, the text grammar model can also be a pre-set text grammar template. This example embodiment does not specifically limit the way to achieve semantic sorting of multiple words. For example, for an image of a child flying a kite on the grass, the image is input into a multi-label classification model for image label extraction. The image labels obtained can be "children", "grass", and "kite". The image labels are input into the text grammar model, and the arrangement order of "children" is 1, that is, "children" is the first of the multiple image labels to appear in the complete sentence. Similarly, the arrangement order of "grass" is 2, and the arrangement order of "kite" is 3. Of course, this is only a schematic example, and this example embodiment does not specifically limit this.
[0104] Connectives refer to words that are interspersed between multiple words to achieve coherent semantics. For example, image labels may be "children", "grass", and "kite". The text grammar model is used to determine that the arrangement order of the image labels should be 1, 2, and 3. The text grammar model is then used to further determine that the connectives between the image labels are: the connectives between the image labels "children" and "grass" may be "in", "running in", etc., and the connectives between the image labels "grass" and "kite" may be "put on", "play on", etc. Of course, this is only an illustrative example, and this example embodiment does not specifically limit this.
[0105] By filtering and matching the target text data through the similarity between the keywords of the text data stored in the database and the image tags, or directly generating the target text data by assembling the image tags, or generating the target text data according to the generated arrangement order, conjunctions and image tags, the matching efficiency of the target text data describing the image data can be effectively improved, while also ensuring the accuracy of the matching results and improving the user experience. In addition, the entire solution has a small amount of computation and can be deployed on mobile terminals with low deployment cost and difficulty, realizing the function of describing the input image data as a logically coherent sentence.
[0106] Furthermore, after determining the target text data corresponding to the image data, the target text data can be played in the form of voice broadcast, for example, the target text data can be broadcasted through TTS (Text To Speech) technology. By playing the target text data in the form of voice broadcast, in specific scenarios (such as the application scenario of children learning to speak by looking at pictures, or the application scenario of the blind listening to pictures), the user's use is more convenient and faster, and the user experience is improved.
[0107] In an exemplary embodiment, when the data description method is deployed to a terminal side with a smaller amount of computation, especially a mobile terminal side with an even smaller amount of computation, due to the limitation of computing performance, it is impossible to realize real-time search of description data and identification processing. Therefore, the description data collected in advance under different target data types can be obtained, and the second feature information corresponding to the description data can be extracted. Then, the description data can be associated with the second feature information and stored in a database. In this way, when matching with the input data, there is no need to search for a large amount of description data and calculate a large amount of feature information of the description data. The extracted first feature information can be directly matched with the second feature information in the database, and the associated description data can be located according to the matched second feature information, so as to realize fast matching of the target description data and improve matching efficiency.
[0108] In an exemplary embodiment, in order to further reduce the data collection cost in the database, after each description of the input data, the input data can be used as new description data, and the first characteristic information can be used as the second characteristic information of the new description data. The new description data and the second characteristic information of the new description data can then be associated and added to the database as new data. This can effectively update and supplement the database, making the subsequent matching more accurate, while effectively reducing the data collection cost.
[0109] In an exemplary embodiment, the Figure 8 The steps in the embodiment are to match the target description data from the description data corresponding to the target data type according to the first feature information and the second feature information, referring to Figure 8 As shown, it may specifically include:
[0110] Step S810, determining a first word vector of the first feature information, and determining a second word vector of the second feature information;
[0111] Step S820, calculating the similarity between the first feature information and the second feature information according to the first word vector and the second word vector;
[0112] Step S830: Match target description data from the description data according to the second feature information whose similarity is greater than or equal to a similarity threshold.
[0113] Among them, the first word vector refers to the word vector (Word embedding) corresponding to the first feature information, and the second word vector refers to the word vector corresponding to the second feature information. For example, the first feature information and the second feature information can be converted into word vectors through the Word2vec model. Of course, the first feature information and the second feature information can also be converted into word vectors through other word vector conversion methods. This example embodiment is not limited to this. Since the first feature information and the second feature information are both words that cannot be understood by the machine, such as keywords or image tags, the calculation amount is large or the accuracy is low when calculating the similarity. Therefore, the first feature information and the second feature information can be converted into word vectors, and the task of measuring the similarity between words is converted into the task of calculating the distance between vectors, which effectively improves the efficiency of the similarity calculation between the first feature information and the second feature information and reduces the calculation amount.
[0114] The similarity threshold refers to pre-set data used to measure whether the similarity between the first feature information and the second feature information meets the matching criteria. For example, the similarity threshold may be 0.8. Assuming that the similarity between the first feature information and the second feature information is 0.9, the description data corresponding to the second feature information may be considered as the target description data. Of course, the similarity threshold may also be 0.7 or 0.9. The specific similarity threshold may be customized according to actual conditions, and this example embodiment does not impose any special limitation on this.
[0115] It should be noted that the "first" and "second" in the "first word vector" and "second word vector" in this example embodiment are only used to distinguish the word vectors of the first feature information and the second feature information. They do not have any special meaning in essence and should not cause any special limitation to this example embodiment.
[0116] In an exemplary embodiment, the input data may include one or more combinations of text data, image data, voice data or music data; Fig. 9 The steps in the embodiment are to match the target description data from the description data corresponding to the target data type according to the first feature information and the second feature information, referring to Fig. 9 As shown, it may specifically include:
[0117] Step S910, extracting emotion type data corresponding to the text data, image data, voice data or music data;
[0118] Step S920: Match target description data from the description data according to the first feature information, the second feature information, and the emotion type data.
[0119] Among them, the emotion type data can be the emotions expressed by the information contained in the text data, image data, voice data or music data. For example, the emotion type data can be surface emotion types such as happy and sad, or deep emotion types such as positive and depressed. This example embodiment does not specifically limit the emotion type corresponding to the data.
[0120] When matching the description data corresponding to the input data, in addition to the similarity between the first feature information and the second feature information, the sentiment type data is introduced to further improve the accuracy of the matching result.
[0121] Specifically, when extracting the emotion type data corresponding to the text data, the emotion identifier in the text data can be extracted. The emotion identifier refers to the key identification data in the input data or description data that can represent the emotion information. For example, the emotion identifier can include one or more combinations of emotion adjectives, modal particles, and punctuation marks; and then the emotion type data of the text data can be determined based on the emotion identifier.
[0122] When extracting the emotion type data corresponding to the image data, the color distribution style corresponding to the image data can be identified, and a preset style-emotion mapping relationship can be obtained, and the emotion type data of the image data can be determined from the style-emotion mapping relationship according to the identified color distribution style.
[0123] When extracting emotion type data corresponding to speech data, a first emotion vector corresponding to the speech data can be extracted. The first emotion vector may include the probability that each speech feature in the speech data is identified as a different emotion type, and then the emotion type data of the speech data can be determined based on the first emotion vector.
[0124] When extracting the emotion type data corresponding to the music data, the audio modal features and the text modal features of the music data may be calculated, and then the emotion type data of the music data may be determined based on the audio modal features and the text modal features.
[0125] In an application scenario of this example embodiment, target description data is determined from the description data corresponding to the target data type based on the first feature information, the second feature information and the emotion type data. This can be implemented as an application or plug-in such as a memo that has a recording function or assists in editing Moments or Weibo. For example, after the user inputs a voice, text, images, music and / or accompaniment with the same description content and emotion type as the voice can be matched, and the matched text, images, music and / or accompaniment can be assembled to quickly record the user's mood.
[0126] For example, a user inputs the voice message "It's a sunny day today, I'm so happy", and it is matched with a text describing a happy mood or positivity, an image with sunny weather or bright colors, and music with a cheerful rhythm. The matched text, image, music and input voice are recorded together as a diary. Of course, this is only an illustrative example and should not impose any special limitations on this example embodiment.
[0127] In summary, in this exemplary embodiment, the first feature information of the input data and at least one target data type used to describe the input data are first determined, and then the second feature information of the description data corresponding to the target data type is obtained. Finally, the target description data is determined from the description data corresponding to the target data type according to the first feature information and the second feature information, and then the input data can be described cross-modally through the target description data. On the one hand, the first feature information of the input data and the second feature information of the description data are extracted, and the target description data is screened and matched through the first feature information and the second feature information, thereby avoiding the information loss generated when the input data is directly converted into the description data, and improving the accuracy of the matched target description data; on the other hand, the conversion between the semantically complex input data and the description data is converted into the matching between the semantically simple first feature information and the second feature information, effectively reducing the amount of calculation, reducing the deployment cost and the deployment difficulty; on the other hand, by actively selecting the target data type used to describe the input data and narrowing the scope of data screening and matching, the amount of calculation can be further reduced, and the efficiency of data matching conversion can be improved. At the same time, by selecting multiple target data types, the target description data finally obtained can be more diverse and the accuracy of the target description data in cross-modal characterization of the input data can be improved.
[0128] It should be noted that the above figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.
[0129] For further reference, Fig.10 As shown, in the implementation of this example, a data description device 1000 is also provided, which may include an input data acquisition module 1010, a first feature information extraction module 1020, a second feature information acquisition module 1030 and a target description data screening module 1040. Among them:
[0130] The input data acquisition module 1010 is used to acquire input data;
[0131] The first feature information extraction module 1020 is used to determine the first feature information of the input data and determine at least one target data type used to describe the input data;
[0132] The second characteristic information acquisition module 1030 is used to acquire the description data corresponding to the target data type and the second characteristic information associated with the description data;
[0133] The target description data screening module 1040 is used to match the target description data from the description data according to the first feature information and the second feature information, so as to describe the input data in a cross-modal representation based on the target description data.
[0134] In an exemplary embodiment, the data description device 1000 may include a mutual conversion description module, and the mutual conversion description module may be used to:
[0135] Extracting keywords corresponding to the text data, and associating the keywords with the text data as description data of the text data type;
[0136] Extracting an image tag corresponding to the image data, and associating the image tag with the image data as description data of the image data type;
[0137] A matching relationship between the description data of the text data type and the description data of the image data type is determined based on the keywords and the image tags, so as to implement a mutual conversion description between the text data and the image data according to the matching relationship.
[0138] In an exemplary embodiment, the first feature information extraction module 1020 may be used to:
[0139] Performing word segmentation processing on the text data to extract keywords of the text data; and
[0140] The image labels of the image data are extracted through a multi-label model.
[0141] In an exemplary embodiment, the data description device 1000 further includes an album search function module, which can be used to:
[0142] Acquire text data input in the photo album application, and determine keywords of the text data;
[0143] Obtaining image tags corresponding to the album images stored in the album application, and calculating the similarity between the keyword and the image tag;
[0144] The target album images are determined from the album images according to the similarity, and the target album images are sorted and displayed according to the similarity.
[0145] In an exemplary embodiment, the data description device 1000 further includes an image description module, which can be used to:
[0146] Acquire image data to be described, and determine an image tag of the image data;
[0147] Acquire keywords of text data stored in a database, calculate the similarity between the image tag and the keyword, and determine target text data from the text data according to the similarity; or
[0148] Generate target text data according to the image label.
[0149] In an exemplary embodiment, the image description module may include an image tag assembly unit, which may be used to:
[0150] Determining the arrangement order and connecting words between the image tags according to a preset text grammar model;
[0151] Target text data for describing the image data is generated through the arrangement order, the connecting words, and the image tags.
[0152] In an exemplary embodiment, the data description device 1000 further includes a description data storage module, which can be used to:
[0153] Acquire pre-collected description data under different target data types, and extract second feature information corresponding to the description data;
[0154] The description data is associated with the second feature information and stored in a database.
[0155] In an exemplary embodiment, the data description device 1000 further includes a description data supplement module, which can be used to:
[0156] Using the input data as new description data, and using the first feature information as second feature information of the new description data;
[0157] The new description data and the second characteristic information of the new description data are associated and stored in a database.
[0158] In an exemplary embodiment, the target description data screening module 1040 may also be used to:
[0159] Determine a first word vector of the first feature information, and determine a second word vector of the second feature information;
[0160] Calculate the similarity between the first feature information and the second feature information according to the first word vector and the second word vector;
[0161] According to the second feature information whose similarity is greater than or equal to a similarity threshold, target description data is matched from the description data.
[0162] In an exemplary embodiment, the input data may include text data, image data, voice data or music data; the target description data screening module 1040 may also include:
[0163] An emotion type data extraction unit, used to extract emotion type data corresponding to the text data, image data, voice data or music data;
[0164] A target description data screening unit is used to match target description data from the description data according to the first feature information, the second feature information and the emotion type data.
[0165] In an exemplary embodiment, the emotion type data extraction unit may also be used to:
[0166] Extracting the emotion identifier from the text data, and determining the emotion type data of the text data according to the emotion identifier, wherein the emotion identifier includes one or more combinations of emotion adjectives, modal particles, and punctuation marks; or
[0167] Identifying a color distribution style corresponding to the image data, and determining the emotion type data of the image data according to a preset style-emotion mapping relationship and the color distribution style; or
[0168] Extracting a first emotion vector corresponding to the speech data, and determining emotion type data of the speech data according to the first emotion vector, wherein the first emotion vector includes probabilities that speech features in the speech data are identified as different emotion types; or
[0169] The audio modal features and the text modal features of the music data are calculated, and the emotion type data of the music data is determined according to the audio modal features and the text modal features.
[0170] The specific details of each module in the above device have been described in detail in the implementation method of the method part. The undisclosed details can be found in the implementation method of the method part, so they will not be repeated here.
[0171] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".
[0172] The exemplary embodiments of the present disclosure also provide a computer-readable storage medium on which a program product capable of implementing the above-mentioned method of the present specification is stored. In some possible implementations, various aspects of the present disclosure can also be implemented in the form of a program product, which includes program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section of the present specification, for example, it can execute Figures 3 to 9 Any one or more steps in .
[0173] It should be noted that the computer-readable medium shown in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.
[0174] In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.
[0175] In addition, program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).
[0176] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.
[0177] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims
1. A data description method, characterized in that: Applied to mobile terminals, including: Get input data; Determine first characteristic information of the input data and determine at least one target data type used to describe the input data; the target data type refers to a data type corresponding to the description data used to describe the input data; Acquire description data corresponding to the target data type and second characteristic information associated with the description data; Matching target description data from the description data according to the first feature information and the second feature information, so as to describe the input data in a cross-modal representation based on the target description data; The description data at least includes text data and image data; the method further includes: Extracting keywords corresponding to the text data, and associating the keywords with the text data as description data of the text data type; Extracting an image tag corresponding to the image data, and associating the image tag with the image data as description data of the image data type; A matching relationship between the description data of the text data type and the description data of the image data type is determined based on the keywords and the image tags, so as to implement a mutual conversion description between the text data and the image data according to the matching relationship.
2. The method according to claim 1, characterized in that The extracting keywords corresponding to the text data includes: Performing word segmentation processing on the text data to extract keywords of the text data; and The extracting the image tag corresponding to the image data includes: The image labels of the image data are extracted through a multi-label model.
3. The method according to claim 1, characterized in that The method further comprises: Acquire text data input in the photo album application, and determine keywords of the text data; Obtaining image tags corresponding to the album images stored in the album application, and calculating the similarity between the keyword and the image tag; The target album images are determined from the album images according to the similarity, and the target album images are sorted and displayed according to the similarity.
4. The method according to claim 1, characterized in that: The method further comprises: Acquire image data to be described, and determine an image tag of the image data; Acquire keywords of text data stored in a database, calculate the similarity between the image tag and the keyword, and determine target text data from the text data according to the similarity; or Generate target text data according to the image label.
5. The method according to claim 4, characterized in that Generating target text data according to the image label includes: Determining the arrangement order and connecting words between the image tags according to a preset text grammar model; Target text data for describing the image data is generated through the arrangement order, the connecting words, and the image tags.
6. The method according to claim 1, characterized in that The method further comprises: Acquire pre-collected description data under different target data types, and extract second feature information corresponding to the description data; The description data is associated with the second feature information and stored in a database.
7. The method according to claim 1, characterized in that The method further comprises: Using the input data as new description data, and using the first feature information as second feature information of the new description data; The new description data and the second characteristic information of the new description data are associated and stored in a database.
8. The method according to claim 1, characterized in that Matching target description data from the description data according to the first feature information and the second feature information includes: Determine a first word vector of the first feature information, and determine a second word vector of the second feature information; Calculate the similarity between the first feature information and the second feature information according to the first word vector and the second word vector; According to the second feature information whose similarity is greater than or equal to a similarity threshold, target description data is determined from the description data.
9. The method according to claim 1, characterized in that: The input data includes text data, image data, voice data or music data; The matching target description data from the description data according to the first feature information and the second feature information includes: Extracting emotion type data corresponding to the text data, image data, voice data or music data; Match target description data from the description data according to the first feature information, the second feature information, and the emotion type data.
10. The method according to claim 9, characterized in that Extracting emotion type data corresponding to the text data, image data, voice data or music data includes: Extracting the emotion identifier from the text data, and determining the emotion type data of the text data according to the emotion identifier, wherein the emotion identifier includes one or more combinations of emotion adjectives, modal particles, and punctuation marks; or Identifying a color distribution style corresponding to the image data, and determining the emotion type data of the image data according to a preset style-emotion mapping relationship and the color distribution style; or Extracting a first emotion vector corresponding to the speech data, and determining emotion type data of the speech data according to the first emotion vector, wherein the first emotion vector includes probabilities that speech features in the speech data are identified as different emotion types; or The audio modal features and the text modal features of the music data are calculated, and the emotion type data of the music data is determined according to the audio modal features and the text modal features.
11. A data description device, characterized in that: include: An input data acquisition module, used for acquiring input data; A first feature information extraction module, used to determine first feature information of the input data and to determine at least one target data type used to describe the input data; the target data type refers to a data type corresponding to the description data used to describe the input data; A second characteristic information acquisition module, used to acquire description data corresponding to the target data type and second characteristic information associated with the description data; a target description data screening module, configured to match target description data from the description data according to the first feature information and the second feature information, so as to describe the input data cross-modally based on the target description data; The description data at least includes text data and image data; the device is also configured to: Extracting keywords corresponding to the text data, and associating the keywords with the text data as description data of the text data type; Extracting an image tag corresponding to the image data, and associating the image tag with the image data as description data of the image data type; A matching relationship between the description data of the text data type and the description data of the image data type is determined based on the keywords and the image tags, so as to implement a mutual conversion description between the text data and the image data according to the matching relationship.
12. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.
13. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 10 by executing the executable instructions.
Citation Information
Patent Citations
Text generation method and device, equipment and client
CN107766349A
Image description method and terminal equipment
CN109657079A
Image retrieval method, image retrieval device and electronic equipment
CN112765387A
