Image processing method and electronic equipment

Abstract

The embodiment of the invention provides an image processing method and electronic equipment, and relates to the technical field of terminals. The method comprises the steps of performing matching according to first semantic feature information of a first image and respective second semantic feature information of a plurality of to-be-selected texts so as to select a plurality of intermediate texts from the plurality of to-be-selected texts; and performing matching according to the first spatio-temporal feature information of the first image and the second spatio-temporal feature information of the plurality of intermediate texts so as to determine a target text in the plurality of intermediate texts. And displaying the target text for the first image. In this way, the image and the to-be-selected text are matched in combination with the time information and the space information, and the situation that the spatio-temporal information corresponding to the target text does not conform to the spatio-temporal information corresponding to the image can be effectively avoided.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to an image processing method and an electronic device. Background Technology

[0002] Thanks to advancements in image processing, more and more terminal devices support the technology of adding captions to images. This imbues images with more information and emotional depth, allowing users to access more content when viewing them.

[0003] In current implementations, some algorithms can extract features from the content of an image, and then determine the corresponding text based on the degree of matching between the image's feature vector and the feature vector of text in a pre-defined text library. In other words, current implementations typically match the content of the image with the content of the text to achieve image-text matching.

[0004] However, when matching text based on image content, there may still be issues where the text and image content do not match. Summary of the Invention

[0005] This application provides an image processing method and an electronic device, applicable to the field of terminal technology. This effectively avoids situations where the spatiotemporal information corresponding to the accompanying text does not match the spatiotemporal information corresponding to the image.

[0006] In a first aspect, embodiments of this application propose an image processing method. The method includes: matching a first semantic feature information of a first image with second semantic feature information of each of a plurality of candidate texts to select a plurality of intermediate texts; matching a first spatiotemporal feature information of the first image with second spatiotemporal feature information of each of the plurality of intermediate texts to determine a target text among the plurality of intermediate texts; and displaying the target text for the first image.

[0007] In this way, by extracting semantic information from both the first image and the candidate text, and matching the semantic information of the first image with that of the candidate text, text with a stronger semantic match to the first image can be obtained, providing a solid data foundation for subsequent processing. Furthermore, by extracting temporal and spatial information from both the first image and the candidate text, and matching the spatiotemporal information of the first image with that of the intermediate text, intermediate text that matches the spatiotemporal information of the first image can be effectively retained, thus enabling more accurate and targeted matching of text to the first image.

[0008] In one possible implementation, matching is performed based on the first spatiotemporal feature information of the first image and the second spatiotemporal feature information of each of the multiple intermediate texts to determine the target text among the multiple intermediate texts. This may include: filtering out intermediate texts from the multiple intermediate texts whose second spatiotemporal feature information does not match the first spatiotemporal feature information; and determining the target text from the remaining intermediate texts after filtering.

[0009] In one possible implementation, the first spatiotemporal feature information includes first temporal feature information and first spatial feature information, and the second spatiotemporal feature information includes second temporal feature information and second spatial feature information. Filtering intermediate texts from multiple intermediate texts where the second spatiotemporal feature information does not match the first spatiotemporal feature information includes: filtering intermediate texts where the second temporal feature information does not match the first temporal feature information, and filtering intermediate texts where the second spatial feature information does not match the first spatial feature information.

[0010] In this way, by filtering the intermediate text based on the first temporal feature information and the first spatial feature information, intermediate text that matches the first image semantically but does not match the first image in terms of spatiotemporal information can be eliminated, thereby providing a data foundation for subsequent processing.

[0011] In one possible implementation, the time feature information includes an encoding consisting of M fields, where M is an integer greater than or equal to 1.

[0012] The M fields include at least one of the following: a first field indicating the season, at least one second field indicating the festival, and a third field indicating the time range.

[0013] In this way, the time-related content contained in the image is extracted based on multiple time information such as season, festival and time range, and the time-related content contained in the text is extracted based on multiple time information, so as to match the image with text that is more consistent with the time information of the image.

[0014] In one possible implementation, filtering intermediate texts from multiple intermediate texts where the second time feature information does not match the first time feature information includes: for any intermediate text, if the second time feature information of the intermediate text does not satisfy any item in the first condition set, then the intermediate text is filtered.

[0015] The first set of conditions includes: The first field in the second time feature information matches the season indicated by the first field in the first time feature information, or the first field in the second time feature information does not indicate a season. The second field in the second time feature information matches the festival indicated by the second field in the first time feature information, or the second field in the second time feature information does not indicate a festival. The first time range indicated by the third field in the second time feature information overlaps with the second time range indicated by the third field in the first time feature information, or the third field in the second time feature information does not indicate a time range.

[0016] In this way, we can match images with text that is closely related to the time information contained in the images, or obtain text that is more general or comprehensive, so as to increase the diversity of text and facilitate practical needs.

[0017] In one possible implementation, the spatial feature information includes an encoding consisting of N fields, where N is an integer greater than or equal to 1.

[0018] Among them, the N fields include at least one of the following: at least one fourth field for indicating the geographical location, and a fifth field for indicating the environment, wherein the geographical location indicated by the fourth field corresponds to a preset geographical level.

[0019] In this way, spatial content contained in the image is extracted based on the spatial information of the location and environment, and spatial content contained in the text is extracted based on multiple spatial information, so as to match the image with text that is more consistent with the spatial information of the image.

[0020] In one possible implementation, filtering intermediate texts from multiple intermediate texts where the second spatial feature information does not match the first spatial feature information includes: for any intermediate text, if the second spatial feature information of the intermediate text does not satisfy any item in the second condition set, then the intermediate text is filtered.

[0021] The second set of conditions includes: the fourth field in the second spatial feature information is consistent with the geographical location indicated by the fourth field in the first spatial feature information, or the fourth field in the second temporal feature information does not indicate a geographical location. The fifth field in the second spatial feature information is consistent with the environment indicated by the fifth field in the first spatial feature information, or the fifth field in the second temporal feature information does not indicate the environment.

[0022] In this way, we can match text closely related to the spatial information contained in the image, or obtain text with strong generality or summarization to increase the diversity of text and facilitate practical needs.

[0023] In one possible implementation, the image processing method further includes: acquiring first time parameters of the first image, and encoding the first time parameters according to a first encoding rule to obtain first time feature information of the first image. The first time parameters include at least one of the following: the season, holiday, and time range corresponding to the capture of the first image.

[0024] Furthermore, for any candidate text, a second time parameter of the candidate text is obtained, and the second time parameter is encoded according to the first encoding rule to obtain the second time feature information of the candidate text. The second time parameter includes at least one of the following: the season, festival, and time range corresponding to the text content of the candidate text.

[0025] In one possible implementation, the image processing method further includes: acquiring a first location parameter of the first image, and encoding the first location parameter according to a second encoding rule to obtain first spatial feature information of the first image. The first location parameter includes at least one of the following: the geographical location corresponding to the capture of the first image, and the surrounding environment.

[0026] Furthermore, for any candidate text, a second positional parameter of the candidate text is obtained, and the second positional parameter is encoded according to a second encoding rule to obtain the second spatial feature information of the candidate text. The second positional parameter includes at least one of the following: the geographical location corresponding to the text content of the candidate text, and its surrounding environment.

[0027] In one possible implementation, obtaining the second time parameter of the candidate text may include: obtaining the second time parameter of the candidate text based on the text features of the candidate text. And obtaining the second position parameter of the candidate text may include: obtaining the second position parameter of the candidate text based on the text features of the candidate text.

[0028] Secondly, embodiments of this application provide an image processing apparatus, which may be an electronic device, or a chip or chip system within an electronic device. The image processing apparatus may include a display unit and a processing unit.

[0029] When the image processing apparatus is an electronic device, the display unit therein can be a display screen. The display unit is used to perform the display step so that the electronic device implements an image processing method described in the first aspect or any possible implementation of the first aspect.

[0030] When the image processing apparatus is an electronic device, the processing unit may be a processor. The image processing apparatus may further include a storage unit, which may be a memory. The storage unit is used to store instructions, and the processing unit executes the instructions stored in the storage unit to cause the electronic device to implement an image processing method described in the first aspect or any possible implementation thereof.

[0031] When the image processing device is a chip or chip system within an electronic device, the processing unit can be a processor. The processing unit executes instructions stored in a storage unit to cause the electronic device to implement an image processing method described in the first aspect or any possible implementation of the first aspect. The storage unit can be a storage unit within the chip (e.g., a register, cache, etc.) or a storage unit located outside the chip within the electronic device (e.g., a read-only memory, random access memory, etc.).

[0032] For example, a processing unit is used to process an image and add text to the image based on an image processing device. A display unit is used to display the image and the text matched to the image.

[0033] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, the memory for storing code instructions, and the processor for running the code instructions to perform the methods described in the first aspect or any possible implementation of the first aspect.

[0034] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program or instructions that, when executed on a computer, cause the computer to perform the methods described in the first aspect or any possible implementation thereof.

[0035] Fifthly, embodiments of this application provide a computer program product including a computer program, which, when run on a computer, causes the computer to perform the methods described in the first aspect or any possible implementation thereof.

[0036] Sixthly, this application provides a chip or chip system including at least one processor and a communication interface. The communication interface and the at least one processor are interconnected via a circuit. The at least one processor is used to run computer programs or instructions to perform the methods described in the first aspect or any possible implementation of the first aspect. The communication interface in the chip can be an input / output interface, pins, or circuits, etc.

[0037] In one possible implementation, the chip or chip system described above in this application further includes at least one memory storing instructions. The memory can be an internal storage unit of the chip, such as a register or cache, or it can be a storage unit of the chip itself (e.g., read-only memory, random access memory, etc.).

[0038] It should be understood that the second to sixth aspects of this application correspond to the technical solutions of the first aspect of this application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation are similar, and will not be repeated here. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of an image processing scenario provided in an embodiment of this application;

[0040] Figure 2 A flowchart illustrating one implementation of the image captioning algorithm provided in this application embodiment;

[0041] Figure 3 This is a schematic diagram of the hardware structure of a terminal device provided in an embodiment of this application;

[0042] Figure 4 This is a schematic diagram of the software structure of a terminal device provided in an embodiment of this application;

[0043] Figure 5 This is a schematic diagram of the image processing process provided in the embodiments of this application;

[0044] Figure 6 A schematic diagram of the structure of time feature information provided in the embodiments of this application;

[0045] Figure 7 A schematic diagram illustrating the encoding of eight time ranges provided in an embodiment of this application;

[0046] Figure 8 A schematic diagram illustrating the process of obtaining first spatiotemporal feature information provided in an embodiment of this application;

[0047] Figure 9 A schematic diagram illustrating eight time ranges provided for embodiments of this application;

[0048] Figure 10 This is a schematic diagram illustrating the process of filtering text based on a first condition set, provided in an embodiment of this application.

[0049] Figure 11 Signaling interaction diagram of the image processing method provided in the embodiments of this application;

[0050] Figure 12 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0051] To facilitate a clear description of the technical solutions in the embodiments of this application, some terms and technologies involved in the embodiments of this application will be briefly introduced below:

[0052] 1. Exif information

[0053] Exif (Exchangeable Image File) information is metadata embedded in an image file. It contains various information such as the image's capture time, geographical location (latitude and longitude), camera model, and exposure parameters. Metadata refers to additional information embedded in the image file; this information is not the image's pixel data itself, but rather descriptive data about the image's content, attributes, shooting conditions, and origin.

[0054] 2. Spatiotemporal information

[0055] Spatiotemporal information can include both spatial and temporal information of an image. Spatial information can include the actual geographical location of objects or scenes within the image, as well as spatial attributes such as the relative position, shape, and size of objects or features. Temporal information can include the specific time the image was captured or recorded, which can be obtained through metadata such as Exif. For time-series images or videos, temporal information also includes changes in image content over time, such as object movement or scene changes.

[0056] 3. POI (Point of Interest)

[0057] In Geographic Information Systems (GIS), a Point of Interest (POI) typically refers to any point on a map that is not geographically significant, such as shops, bars, gas stations, hospitals, or train stations—points that can attract people's interest or demand. POI data usually includes attributes such as the location's name, address, coordinates (longitude and latitude), and category.

[0058] 4. Other terms

[0059] In the embodiments of this application, terms such as "first" and "second" are used to distinguish identical or similar items with substantially the same function and purpose. For example, "first chip" and "second chip" are used only to distinguish different chips and do not limit their order of execution. Those skilled in the art will understand that terms such as "first" and "second" do not limit the quantity or execution order, and that "first" and "second" do not necessarily imply that they are different.

[0060] It should be noted that, in the embodiments of this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design scheme described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0061] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, a--c, bc, or abc, where a, b, and c can be single or multiple.

[0062] 5. Electronic equipment

[0063] The electronic devices in this application embodiment may include handheld devices with image processing functions, vehicle-mounted devices, etc. For example, some electronic devices include: mobile phones, tablets, PDAs, laptops, mobile internet devices (MIDs), wearable devices, virtual reality (VR) devices, augmented reality (AR) devices, wireless terminals in industrial control, wireless terminals in self-driving vehicles, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, cellular phones, cordless phones, session initiation protocol (SIP) phones, wireless local loop (WLL) stations, personal digital assistants (PDAs), handheld devices with wireless communication capabilities, computing devices or other processing devices connected to a wireless modem, in-vehicle devices, wearable devices, terminal devices in 5G networks, or future evolution of public land mobile communication networks. Terminal devices in a network (PLMN), etc., are not limited to this in the embodiments of this application.

[0064] By way of example and not limitation, in this embodiment, the electronic device can also be a wearable device. Wearable devices, also known as wearable smart devices, are a general term for devices that utilize wearable technology to intelligently design and develop everyday wearables, such as glasses, gloves, watches, clothing, and shoes. Wearable devices are portable devices that are worn directly on the body or integrated into the user's clothing or accessories. Wearable devices are not merely hardware devices, but also achieve powerful functions through software support, data interaction, and cloud interaction. Broadly speaking, wearable smart devices include those that are feature-rich, large in size, and can achieve complete or partial functions without relying on a smartphone, such as smartwatches or smart glasses, as well as those that focus on a specific type of application function and require the use of other devices such as smartphones, such as various smart bracelets and smart jewelry for vital sign monitoring.

[0065] Furthermore, in this embodiment of the application, the electronic device can also be a terminal device in the Internet of Things (IoT) system. IoT is an important part of the future development of information technology. Its main technical feature is to connect objects to the network through communication technology, thereby realizing an intelligent network of human-machine interconnection and object-to-object interconnection.

[0066] The electronic devices in the embodiments of this application may also be referred to as: terminal equipment, user equipment (UE), mobile station (MS), mobile terminal (MT), access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent, or user device, etc.

[0067] In this embodiment, the electronic device or various network devices include a hardware layer, an operating system layer running on top of the hardware layer, and an application layer running on top of the operating system layer. The hardware layer includes hardware such as a central processing unit (CPU), a memory management unit (MMU), and memory (also called main memory). The operating system can be any one or more computer operating systems that implement business processing through processes, such as Linux, Unix, Android, iOS, or Windows. The application layer includes applications such as browsers, address books, word processing software, and instant messaging software.

[0068] To better understand the technical solution of this application, the relevant technologies involved in this application will be further described in detail below.

[0069] With the advancement of technology and the widespread use of digital devices, more and more users are able to capture beautiful moments in life by taking pictures or videos. Currently, many photo library apps on various devices allow users to add text to their photos, generating a corresponding image with text to share their daily lives with friends. However, manually editing images and writing captions is often time-consuming and laborious. Therefore, the emergence of automatic image captioning can effectively improve the efficiency of image captioning, allowing users to complete this process more easily.

[0070] It is understandable that this image processing method can automatically generate text from images. Therefore, in addition to the application scenarios described above, the method of this application can also be applied to other scenarios that require image captioning. For example, there are the following two specific application scenarios:

[0071] One scenario involves embedding text generated from images in the gallery into the images themselves, displaying it as an image containing text on the terminal device. Another scenario involves displaying the text generated from images in the gallery and the image itself as two separate objects within relevant controls in the gallery app. Specifically, in controls such as "On This Day" or "Recommended" in a gallery app, the text can be displayed as a separate object above or near the image. These controls are designed to recommend or remind users of images.

[0072] The following is combined Figure 1 A brief explanation of the two image processing scenarios described above. Figure 1 This is a schematic diagram of an image processing scenario provided in an embodiment of this application.

[0073] like Figure 1 As shown in the figure, the two application scenarios mentioned above are illustrated. (a) shows the first scenario, and (b) shows the second scenario.

[0074] refer to Figure 1 In (a) of the image library, 103 can be understood as the image display page of the mobile phone. Suppose there is an image 101, and it is necessary to add text to the image 101. The image library can first match a text "The evening breeze is gentle, and the roses are in bloom" based on the information of the image 101. Then, this text can be embedded into the image 101 to generate an image 102 containing text and display it.

[0075] refer to Figure 1 In (b), 104 can be understood as the phone's gallery page, and 105 can be understood as the phone's image display page. The gallery page 104 can contain 8 images, as well as controls such as "Recommended" and "On This Day in History".

[0076] You can refer to this. Figure 1 (b) in the text refers to the second scenario, where image 101 and its caption "A gentle evening breeze, roses in bloom" can be displayed in the content of the "Recommended" control in the image library, and image 106 and its caption "Dancing dragon prays for good fortune" can be displayed in the content of the "On This Day" control in the image library. In this case, image 101 and its caption displayed in the "Recommended" control are two separate objects, and image 106 and its caption displayed in the "On This Day" control are also two separate objects.

[0077] When you click on the "Recommended" or "On This Day" control, you will be redirected from the gallery page 104 to the image display page 105, where the content of the "Recommended" or "On This Day" control will be displayed. For example, clicking on the "Recommended" control will then display image 101 on image display page 105. At this time, image 101 will not display its accompanying text because the image and text are two separate objects.

[0078] However, it should be understood that the image processing method provided in this application is not limited to the two application scenarios mentioned above. The technical solution of this application can be applied to any scenario where captions are added to images.

[0079] Next, we will explain in detail how this image processing method achieves automatic image captioning. In one implementation, the target text for the image to be processed can be selected from multiple texts in a pre-set text library. This method may include the following steps: First, obtain the feature vectors of the image to be processed and the multiple texts in the pre-set text library. Then, match the image feature vector of the image to be processed with the text feature vectors of each text in the pre-set text library to obtain the degree of matching between each text and the image to be processed. Finally, based on the degree of matching between each text and the image to be processed, and according to specific rules, determine the target text for the image to be processed from the multiple texts.

[0080] In one implementation, text that closely matches the image to be processed can be used as the target text for the image to be processed.

[0081] In one implementation, multiple candidate texts can be selected from multiple texts in a preset text library. Then, after further processing, candidate texts that do not match the image to be processed are filtered out. Finally, the target text of the image to be processed is determined from the filtered candidate texts.

[0082] For example, the image to be processed and multiple texts can first be processed through a feature extraction module to obtain the feature vector of the image to be processed and the feature vectors of each of the multiple texts in a preset text library. Then, the matching degree between the image feature vector and the feature vectors of the multiple texts can be calculated, and the texts with higher matching degrees can be selected as candidate texts for the image to be processed. Next, the preset attribute information of the candidate texts, such as scene, sentiment (e.g., emotional tone), and granularity (e.g., the generality of the text), is matched again with the content of the image to filter out mismatched candidate texts, and finally, the filtered text is determined as the target text for the image to be processed. The following section combines... Figure 2 This paper provides a detailed introduction to one implementation method of image captioning algorithm. Figure 2This is a flowchart illustrating one implementation of the image captioning algorithm provided in this application embodiment.

[0083] refer to Figure 2 First, the image to be processed and multiple texts from a pre-defined text library can be input into the feature extraction module. The feature extraction module can then output the feature vectors of the image to be processed and the feature vectors of each of the multiple texts. In one implementation, the feature extraction module can be a model.

[0084] During the training process of this model, the input consists of positive and negative samples. Positive samples are image-text pairs that semantically match; for example, an image of a cat and text describing a cat constitute a positive sample. Negative samples are image-text pairs that semantically do not match; for example, an image of a cat and text describing a dog constitute a negative sample. The model's loss function can be a contrastive learning loss, which measures the model's performance in distinguishing between positive and negative samples. The goal of model training is to maximize the similarity between the image-text feature vectors of positive samples and minimize the similarity between the image-text feature vectors of negative samples. One implementation method uses cosine similarity to calculate the similarity between image-text feature vectors.

[0085] After obtaining the feature vectors of the image to be processed and the feature vectors of each of the multiple texts, refer to Figure 2 This method allows for the calculation of the matching degree between image feature vectors and multiple text feature vectors. One implementation uses cosine similarity to calculate the matching degree between text feature vectors and image feature vectors. After obtaining the matching degrees of multiple text feature vectors corresponding to image feature vectors, the matching degrees can be sorted in descending order.

[0086] In one implementation, text with a high degree of matching can be used as the target text for the image to be processed. In another implementation, a predetermined number of candidate texts with a high degree of similarity to the image feature vector of the image to be processed can be selected first. Then, based on predetermined attributes such as scene, sentiment, and granularity, the candidate texts are matched with the image again. Finally, the text that matches the image to be processed is determined from the multiple candidate texts.

[0087] In summary, existing technologies can achieve image-text matching by semantically matching images and text. However, due to factors such as the similarity of visual features and technological limitations, there may be a mismatch between the text content and the image content. Specifically, this may lead to inconsistencies between the spatiotemporal information of the final selected target text and the spatiotemporal information of the image to be processed.

[0088] The following example illustrates this problem in detail. Suppose a user takes a picture of a sunset. Visually, this image might share some similarities with an image of a sunrise. For instance, the sky may display colors like red, orange, or purple at sunrise and sunset, the sunlight may become softer and warmer, and the shapes and textures of the clouds may also appear similar. In this situation, feature extraction and scene detection methods may struggle to accurately extract the key features that distinguish the sunset / sunset image from the sunrise / sunset image. This makes it difficult to accurately differentiate between the two. Therefore, when matching text to this sunset / sunset image, it's possible to match text related to the sunrise / sunset image.

[0089] Similarly, it is understandable that due to factors such as the similarity of visual features and technical limitations, the spatial information of an image may not be accurately identified, which may lead to a mismatch between the spatial information corresponding to the target text and the spatial information corresponding to the image to be processed.

[0090] Therefore, it is understandable that in the related technologies described above, the selection of target text is affected by the spatiotemporal information of the image, which may lead to inaccurate image captions and thus affect the overall performance of the image captioning system.

[0091] To address the problems described above, this application proposes the following technical concept: First, semantic matching can be performed between the image and the text to filter out multiple texts that have a high degree of semantic matching with the image. Then, the spatiotemporal information of the image and the text can be extracted, and the spatiotemporal information contained in the filtered text can be matched with the spatiotemporal information contained in the image to eliminate texts that do not match the spatiotemporal information contained in the image. Finally, the target text of the image is selected from the remaining text.

[0092] The image processing method of this application embodiment can be executed by an electronic device equipped with image processing capabilities, or by a chip, chip system, or processor that supports the implementation of the image processing method by the electronic device. It can also be executed by a logic module or software capable of implementing all or part of the functions of the electronic device; this application does not impose specific limitations in this regard. The image processing method of this application embodiment will be described in detail below using an electronic device as the execution subject as an example.

[0093] Electronic devices can be, for example, terminal devices. The following section will first combine... Figure 3 and Figure 4 A brief introduction to the terminal equipment.

[0094] For example, Figure 3 This is a schematic diagram of the hardware structure of a terminal device provided in an embodiment of this application.

[0095] Figure 3 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. The terminal device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc.

[0096] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the terminal device. In other embodiments of this application, the terminal device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0097] Processor 110 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors. In one implementation, for example, the image processing method provided in this application may be executed by processor 110.

[0098] Display screen 194 is used to display images, videos, etc. For example, images and text matching the images can be displayed on display screen 194.

[0099] The software system of a terminal device can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture, etc. This application uses the layered architecture Android system as an example to illustrate the software structure of the terminal device.

[0100] For example, Figure 4 This is a schematic diagram of the software structure of a terminal device provided in an embodiment of this application.

[0101] like Figure 4 As shown, the layered architecture divides the software into several layers, each with a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the system may include an application layer, an application framework layer, an Android runtime and system libraries, a hardware abstraction layer (HAL), and a kernel layer. It should be noted that this application uses the Android system as an example; however, the solution can also be implemented in other operating systems (such as HarmonyOS, iOS, etc.) as long as the functions implemented by each module are similar to those in the embodiments of this application.

[0102] The application layer can include a series of application packages.

[0103] like Figure 4 As shown, the application package may include applications such as gallery, camera, calendar, phone, map, music, settings, email, video, and social media. Of course, the application layer may also include other application packages, such as third-party applications like payment apps, shopping apps, banking apps, and social media apps; this application is not limited to these. In this embodiment, the gallery in the application layer can call the management module to implement related functions. The management module may include an image management module and a text management module. For example, the image management module can analyze images to obtain image feature information, such as semantic and spatiotemporal information. Similarly, the text management module can analyze text to obtain text feature information, such as semantic and spatiotemporal information.

[0104] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0105] like Figure 4As shown, the application framework layer may include a window manager, content provider, resource manager, view system, notification manager, etc. In this embodiment, the application framework layer may at least deploy a parsing module and an encoding module. For example, the parsing module may include an image parsing module and a semantic parsing module, and the encoding module may include an image encoding module and a text encoding module. For example, an image can be input to the image parsing module to obtain the image's temporal and spatial information. Then, the image's spatiotemporal information (time and spatial information) is input to the image encoding module, which can output the encoding of the image's spatiotemporal information. Similarly, text can be input to the text encoding module to output the encoding of the text's spatiotemporal information. Furthermore, an image can be input to the semantic parsing module to obtain the image's semantic feature vector, and text can be input to the semantic parsing module to obtain the text's semantic feature vector.

[0106] The Android runtime consists of the core libraries and the virtual machine. The Android runtime is responsible for the scheduling and management of the Android system.

[0107] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

[0108] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0109] The system library can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), etc.

[0110] The HAL layer is a wrapper around Linux kernel drivers, providing interfaces to the upper layers and shielding them from the implementation details of the lower-level hardware.

[0111] The HAL layer may include a Wi-Fi HAL, an audio HAL, a Camera HALServer unit, and software code libraries. In this embodiment, the HAL layer may at least have a semantic model deployed. For example, images can be input into the semantic model to obtain the semantics of the images, and text can be input into the semantic model to obtain the semantics of the text.

[0112] The kernel layer is the layer between hardware and software. The kernel layer contains at least the display driver, camera driver, audio driver, and sensor driver.

[0113] The technical solutions of the embodiments of this application and how the technical solutions of the embodiments of this application solve the above-mentioned technical problems will be described in detail below with reference to the accompanying drawings and specific examples. The following specific embodiments can be implemented independently or in combination with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0114] The technical solution of this application relates to image processing methods, which will be described below. The image processing process will be explained in conjunction with specific embodiments. Figure 5 The image processing process will be described in detail. Figure 5 This is a schematic diagram illustrating the image processing process provided in an embodiment of this application. The image processing flow may include, for example, the following steps:

[0115] S501, Obtain the text library.

[0116] Before processing the first image, a text library needs to be constructed. The text library can contain multiple candidate texts. In one implementation, the candidate texts in the text library can be manually added, or they can be obtained from the internet. The sources of the candidate texts in the text library can be, but are not limited to, news articles, web pages, and other articles. This embodiment does not restrict the source of the candidate texts. When adding text to the first image, the target text that matches the image can be selected from the multiple candidate texts in the text library.

[0117] S502. Encode the first image and the candidate text to obtain the spatiotemporal feature information of the image and the candidate text respectively.

[0118] Based on the above, it can be understood that the degree of matching between the first image and the candidate text can be measured based on their respective semantic feature information. Here, the semantic feature vector refers to the feature vector mentioned above; specifically, the semantic feature information of the first image refers to its feature vector, and the semantic feature information of the candidate text refers to its feature vector. However, this matching process does not consider the temporal and spatial information of the first image and the candidate text, which may lead to a mismatch between the selected target text and the first image.

[0119] The time information can include seasons, holidays, and specific times of day, such as morning, noon, and afternoon; while the spatial information can include the state, country, city, and a broader description of the geographical location and environment, such as the Northern and Southern Hemispheres, North and South, domestic and foreign, indoor and outdoor.

[0120] Therefore, in order to achieve more accurate text matching for images, this embodiment considers the spatiotemporal information (time information and spatial information) of images and text as a reference for measuring the degree of image-text matching.

[0121] In one implementation, the spatiotemporal information contained in the image can be encoded according to a preset encoding rule to obtain the spatiotemporal feature information (temporal and spatial feature information) of the image. Similarly, the spatiotemporal information contained in the text can be encoded according to a preset encoding rule to obtain the spatiotemporal feature information of the text. Then, based on the degree of matching between the spatiotemporal feature information of the image and the spatiotemporal feature information of the text, the target text that matches the image can be determined.

[0122] S503. Match the first semantic feature information of the first image and the second semantic feature information of each of the multiple candidate texts to select multiple intermediate texts from the multiple candidate texts.

[0123] Based on the above, it can be understood that the process of adding captions to images can include: first, obtaining text that semantically matches the image, and then filtering or processing these texts to obtain target text that matches the image.

[0124] In this embodiment, the semantic feature information of the first image and the candidate text can be matched first to obtain multiple intermediate texts. Then, the spatiotemporal feature information of the first image and the candidate text can be matched to filter out intermediate texts that do not match the spatiotemporal feature information of the first image. Finally, the target text of the first image can be selected from the filtered intermediate texts according to certain rules.

[0125] You can refer to this. Figure 5 To understand the semantic features, the first image is input into the semantic parsing module and the semantic model to obtain the first semantic feature information of the first image. Similarly, the candidate texts are input into the semantic parsing module and the semantic model to obtain the second semantic feature information corresponding to each candidate text. Then, a semantic matching algorithm is used to match the first semantic feature information of the first image with the second semantic feature information of the candidate texts.

[0126] In one implementation, cosine similarity can be used to measure the degree of matching between the first semantic feature information of the first image and the second semantic feature information of the candidate text. Specifically, the greater the cosine similarity between the first semantic feature information of the first image and the second semantic feature information of the candidate text, the higher the degree of matching between the first semantic feature information of the first image and the second semantic feature information of the candidate text.

[0127] In one implementation, after determining the degree of matching between the first semantic feature information of the first image and the second semantic feature information of multiple candidate texts in the text library, multiple intermediate texts can be selected based on the degree of matching. The second semantic feature information of these intermediate texts has a high degree of matching with the first semantic feature information of the first image, thus providing a solid data foundation for subsequent processing.

[0128] For example, the semantic matching algorithm may include: first calculating the cosine similarity between the semantic feature information of the first image and the semantic feature information of the candidate texts; then sorting the cosine similarities in descending order; and finally selecting a predetermined number of candidate texts with the highest cosine similarity in the sort as intermediate texts. For example, the candidate texts with the highest cosine similarity in the sort can be selected as intermediate texts.

[0129] S504. Match the first spatiotemporal feature information of the first image and the second spatiotemporal feature information of each of the multiple intermediate texts to filter out the intermediate texts that do not match the first spatiotemporal feature information.

[0130] In this embodiment, the temporal and spatial feature information of the first image and the multiple intermediate texts can be matched respectively to filter out intermediate texts that do not match the first temporal or spatial feature information of the first image.

[0131] You can refer to this. Figure 5 The process involves matching the first temporal feature information of the first image with the second temporal feature information of multiple intermediate texts, filtering out intermediate texts whose second temporal feature information does not match the first temporal feature information. Similarly, the process involves matching the first spatial feature information of the first image with the second spatial feature information of multiple intermediate texts, filtering out intermediate texts whose second spatial feature information does not match the first spatial feature information.

[0132] S505. Determine the target text from the remaining intermediate text after filtering.

[0133] S506. Display the target text for the first image.

[0134] The following explanation addresses S505 and S506 together. After spatiotemporal coding matching, the number of remaining intermediate texts can be categorized into three cases.

[0135] Scenario 1: There may be an intermediate text that meets the conditions.

[0136] In this case, the intermediate text that meets the conditions can be used as the target text. And for the first image, the target text is displayed.

[0137] Scenario 2: There may be multiple intermediate texts that meet the conditions.

[0138] In this case, one intermediate text can be selected as the target text. In one implementation, one intermediate text can be randomly selected from multiple intermediate texts as the target text, and then the target text is displayed for the first image. In another implementation, one intermediate text can be selected from multiple intermediate texts according to specific rules as the target text, and then the target text is displayed for the first image.

[0139] In one implementation, this specific rule could be to select the intermediate text among the remaining filtered intermediate texts that matches the first image most closely, as the target text. In another implementation, this specific rule could also be to select the intermediate text among the remaining filtered intermediate texts that contains the richest spatiotemporal information, as the target text.

[0140] Scenario 3: There may not be any intermediate text that meets the conditions.

[0141] In this scenario, one implementation can obtain more intermediate text by reducing the semantic matching rules (i.e., increasing the number of intermediate texts). Then, based on the first and second condition sets, the first spatiotemporal feature information of the first image is matched with the second spatiotemporal feature information of the intermediate texts to filter out the intermediate texts that meet the conditions. Finally, the target text is determined from the filtered intermediate texts.

[0142] For example, suppose the current semantic matching rule is to select the candidate texts whose cosine similarity ranks in the top 20 as intermediate texts. However, after the encoding matching algorithm, no intermediate texts meet the criteria. In this case, we can return to the process of determining the intermediate texts from multiple candidate texts, selecting the candidate texts whose cosine similarity ranks in the top 40 as intermediate texts, and then performing subsequent processing.

[0143] This process begins by extracting semantic information from the first image and then from the candidate text. Next, the semantic information of the first image is matched with that of the candidate text to obtain text that highly matches the first image semantically, providing a solid foundation for subsequent processing. Furthermore, temporal and spatial information is extracted from the first image, and similarly, the candidate text is also extracted accordingly. By matching the spatiotemporal information of the first image with that of the candidate text, candidate text that matches the spatiotemporal information of the first image can be effectively filtered out, resulting in more precise and targeted text matching for the first image.

[0144] Based on the above-described embodiments, the following is combined with... Figure 6 and Figure 7 The spatiotemporal coding rules will then be explained in detail. Among them, Figure 6 This is a schematic diagram of the structure of the time feature information provided in the embodiments of this application. Figure 7 This is a schematic diagram of the eight time range codes provided in the embodiments of this application.

[0145] As can be seen from the above description of S502, this embodiment requires encoding images or text based on specific encoding rules to obtain spatiotemporal feature information. Therefore, the implementation of the encoding rules will be explained below.

[0146] In this embodiment, the spatiotemporal coding rules may include time coding rules and spatial coding rules. The time coding rules will be introduced first:

[0147] In one implementation, an image can be encoded based on multiple time parameters to obtain temporal feature information, and similarly, text can be encoded based on multiple time parameters to obtain temporal feature information. The time parameters can indicate the time information of the image or text; for example, an image can contain multiple time parameters, and a piece of text can also contain multiple time parameters. When encoding the time parameters, for example, the encoding result for each time parameter can be considered as a field of the temporal feature information. Then, the encoding results corresponding to each of the multiple time parameters can constitute the temporal feature information. Therefore, in this embodiment, the temporal feature information may, for example, include an encoding consisting of M fields, where M is an integer greater than or equal to 1.

[0148] In this embodiment, the time parameters may include, for example, three items: season, holiday, and time range. The time range may include, for example, specific time periods within a day that are frequently used: midnight, night, early morning, sunrise, morning, noon, afternoon, and sunset. The selection of time parameters and time ranges can be adjusted according to actual needs, and this embodiment does not impose any restrictions on this. The time feature information may, for example, include three fields: a field indicating the season, a field indicating the holiday, and a field indicating the time range.

[0149] In one implementation, each time parameter can be encoded using at least one bit. The encoding rules for the three fields in the time feature information (i.e., the rules for encoding the three time parameters: season, holiday, and time range) are described in detail below:

[0150] For example, when encoding seasons, to distinguish between the four seasons of spring, summer, autumn, and winter, four different values ​​can be set to represent these four seasons respectively. For instance, 1, 2, 3, and 4 can be used to represent spring, summer, autumn, and winter respectively.

[0151] In one implementation, for example, four bits can be used to encode the seasons. For instance, the codes “0001”, “0010”, “0011”, and “0100” can be used to represent the four seasons of spring, summer, autumn, and winter, respectively.

[0152] When coding festivals, it's important to consider the distinction between lunar and solar calendar festivals, and the possibility that a single day may fall on both dates. Therefore, lunar and solar festivals can be coded separately. Specifically, based on a specific calendar or cultural context, a list of lunar festivals for the year can be compiled, and each festival can be assigned a unique code. Similarly, a list of solar festivals for the year can be compiled based on a specific calendar or cultural context, and each solar festival can be assigned a unique code.

[0153] In one implementation, for example, 8 bits can be used to encode the festival, with the first 4 bits indicating the Gregorian calendar festival and the last 4 bits indicating the lunar calendar festival. For example, the code "00100000" can be used to represent Valentine's Day, and the code "00000010" can be used to represent the Lantern Festival. The code "00100010" can be used to represent both Valentine's Day and the Lantern Festival.

[0154] When encoding time ranges, it's understandable that a single text might describe a scenario encompassing one or more time ranges. For example, suppose we have the text "Working at sunrise, resting at sunset; day after day, grateful for every sunrise and sunset." Based on the text's content, we can determine that it describes two scenarios: working at sunrise and resting at sunset. Therefore, considering the eight time ranges mentioned above, the text includes the time ranges of "sunrise" and "sunset."

[0155] Therefore, when encoding time ranges, it is necessary to consider encoding any single time range and any combination of time ranges. One approach is to enumerate all cases (i.e., containing only one time range or multiple time ranges) and assign a unique code to each case. However, this method complicates the subsequent setting of matching rules for images and text. The following section describes this problem with specific examples.

[0156] Suppose we have a sunrise image 'a' and two candidate texts, text 1 and text 2. Text 1 reads, "The sun rises in the east, shining brightly; dawn awakens us, the morning glow intoxicates us," and text 2 reads, "Work at sunrise, rest at sunset; day after day, thank you for accompanying us through every sunrise and sunset." Based on the content of the two texts, we can determine that text 1 describes a sunrise scene, while text 2 describes both sunrise and sunset scenes. Since both texts contain descriptions of sunrise, both texts can be matched with the sunrise image 'a'.

[0157] Assuming we enumerate all possibilities and assign a unique code to each possibility when encoding time ranges, we can assign three different values ​​to the three time ranges: "sunrise," "sunset," and "sunrise and sunset." For example, we could encode "sunrise" as "1," "sunset" as "3," and "sunrise and sunset" as "6." In this case, image a's time range is encoded as "1," text 1's time range is encoded as "1," and text 2's time range is encoded as "6." When matching images and text, image a with a time range encoding of "1" needs to match both candidate texts with time range encodings of "1" and "6," making the matching rule setup relatively complex.

[0158] In one implementation, for example, 8 bits can be used to encode a time range, where each bit represents one of the 8 time ranges. A single bit can have two values: "1" indicates that the image or text contains information about that time range, and "0" indicates that the image or text does not contain information about that time range. In this case, if encoding a combination of multiple time ranges is required, simply setting the corresponding bit for that time range to 1 is sufficient. This eliminates the need to pre-code each combination of time ranges, effectively avoiding the problem of complex matching rules.

[0159] refer to Figure 7 For example, the code for midnight can be set to "00000001", the code for nighttime to "00000010", the code for early morning to "00000100", the code for sunrise to "00001000", the code for morning to "00010000", the code for noon to "00100000", the code for afternoon to "01000000", and the code for sunset to "10000000".

[0160] It is understandable that, based on this encoding method, the time range encoding of sunrise image a in the above example can be "00001000", the time range encoding of text 1 can be "00001000", and the time range encoding of text 2 can be "10001000".

[0161] When matching images and text, a bitwise AND operation can be performed on the time range encodings of the images and text. The result of matching image a with text 1 is "00001000", and the result of matching image a with text 2 is also "00001000". Since their results are not all "0", meaning there are matching bits, the third field of the time feature information in image a matches the third field of the time feature information in both text 1 and text 2.

[0162] Based on the above analysis, it can be understood that 20 bits can be used to encode the three time parameters: season, holiday, and time range. In other words, the time feature information can be composed of 20 bits. (See reference...) Figure 6 To understand this, the 20 bits can be divided into three fields: the first field indicates the season, the second field indicates the holiday, and the third field indicates the time range.

[0163] refer to Figure 6The first field can include 4 bits, the second field can include 8 bits, and the third field can include 8 bits. The encoding of these 20 bits can be found in the above content.

[0164] It can be understood that these 20 bits can be equivalently represented as 5 hexadecimal digits. In the following description, all 20 bits of the time feature information will be written as 5 hexadecimal digits. Therefore, it can be understood that in this embodiment, the first field of the time feature information can contain 1 hexadecimal digit, the second field can contain 2 hexadecimal digits, which can be regarded as the Gregorian calendar digit and the lunar calendar digit respectively, and the third field can contain 2 hexadecimal digits.

[0165] For example, the code for spring can be represented as "0x10000", the code for summer can be represented as "0x20000", the code for autumn can be represented as "0x30000", the code for winter can be represented as "0x40000", the code for Valentine's Day can be represented as "0x02000", the code for the Lantern Festival can be represented as "0x00200", the code for midnight can be represented as "0x00001", the code for night can be represented as "0x00002", the code for early morning can be represented as "0x00004", the code for sunrise can be represented as "0x00008", the code for morning can be represented as "0x00010", the code for noon can be represented as "0x00020", the code for afternoon can be represented as "0x00040", and the code for sunset can be represented as "0x00080".

[0166] For the examples above, the time feature information of sunrise image 'a' can be represented as "0x00008", the time feature information of text 1 can be represented as "0x00008", and the time feature information of text 2 can be represented as "0x00088". These 20 bits are an example of time feature information. The number of bits used to encode time feature information can be adjusted according to actual needs, as long as the encoding of each field can express the meaning of that field. This embodiment does not impose any restrictions on this.

[0167] After detailing the time encoding rules, the spatial encoding rules will now be introduced:

[0168] In one implementation, spatial feature information of an image can be obtained by encoding multiple positional parameters, and spatial feature information of text can be obtained by encoding multiple positional parameters. Positional parameters can be used to indicate the spatial information of an image or text. For example, an image can contain multiple positional parameters, and a piece of text can also contain multiple positional parameters. When encoding the positional parameters, for example, the encoding result of each positional parameter can be regarded as a field of spatial feature information. Then, the encoding results corresponding to multiple positional parameters can constitute the spatial feature information. Therefore, in this embodiment, the spatial feature information may, for example, include an encoding consisting of N fields, where N is an integer greater than or equal to 1.

[0169] In this embodiment, location parameters may include, for example, the geographical location and environment. Multiple geographical levels are predefined for the geographical location, such as state, country, city, Southern or Northern Hemisphere, South or North, and domestic or international. Based on these geographical levels, geographical location information can be extracted from images or text. Furthermore, the environment in the location parameters can include environmental information such as whether it is indoors or outdoors, or the location of a tourist attraction.

[0170] Encoding the location parameters yields spatial feature information containing two fields: a fourth field indicating the geographical location and a fifth field indicating the surrounding environment. The geographical location can include, for example, six preset geographical levels: state, country, city, Southern or Northern Hemisphere, South or North, and domestic or international. The selection of location parameters, preset geographical levels, and the surrounding environment can be adjusted according to actual needs; this embodiment does not impose any restrictions on this.

[0171] In one implementation, each location parameter can be encoded using at least one bit. The encoding rules for the two fields in the spatial feature information (i.e., the rules for encoding the location and environment) are described below:

[0172] When encoding a geographic location, two bits can be used to encode it for three preset geographic levels: Southern or Northern Hemisphere, South or North, and Domestic or International. Four bits can be used to encode it for three preset geographic levels: State, Country, and City.

[0173] When encoding based on the environment, for example, indoor or outdoor environmental information can be encoded using 2 bits. However, for the environmental information of the location of a tourist attraction, 4 bits can be used for encoding.

[0174] For example, 24 bits can be used to encode the two location parameters: geographic location and surrounding environment. That is, spatial feature information can be composed of 24 bits. The fourth field can include 18 bits, and the fifth field can include 6 bits. For instance, suppose there is an image containing the location parameters: Northern Hemisphere, North, China, Indoor. Based on the above encoding rules, the encoding of the fourth field can be determined as "0000 0000 0000101001". The current encoding can be understood by referring to Table 1 below.

[0175] Table 1:

[0176] meaning state nation City Northern and Southern Hemispheres North and South Domestic and international Position parameters / / / Northern Hemisphere north domestic Field value 0000 0000 0000 10 10 01

[0177] Referring to Table 1 above, the field value for "State" is "0000," indicating that the image does not contain information about the relevant state. The field value for "Country" is "0000," indicating that the image does not contain information about the relevant country. The field value for "City" is "0000," indicating that the image does not contain information about the relevant city. The field value for "Northern and Southern Hemispheres" is "10," indicating that the location parameter of the image includes the Northern Hemisphere. The field value for "North and South" is "10," indicating that the location parameter of the image includes the North. The field value for "Domestic and International" is "01," indicating that the location parameter of the image includes the domestic region.

[0178] Additionally, the fifth field can be encoded as “010000”, where “01” represents “indoor” and “0000” means that the image does not contain information about the relevant attractions.

[0179] These 24 bits are an example of spatial feature information. The number of bits used to encode spatial feature information can be adjusted according to actual needs, as long as the encoding of each field can express the meaning of that field. This embodiment does not impose any restrictions on this.

[0180] After obtaining temporal and spatial feature information, they can be fused to obtain spatiotemporal feature information. It is understood that temporal and spatial feature information do not interfere with each other; therefore, in one implementation, temporal and spatial feature information can be directly concatenated to obtain spatiotemporal feature information.

[0181] In this way, the spatiotemporal information contained in the first image and the spatiotemporal information contained in the candidate text are represented as numerical values ​​through encoding, so as to facilitate matching of the spatiotemporal information contained in the first image and the spatiotemporal information contained in the candidate text.

[0182] Based on the above-described embodiments, the following is combined with... Figure 8 and Figure 9 The process of acquiring the first spatiotemporal feature information of the first image will then be described in detail. Figure 8 This is a schematic diagram illustrating the process of obtaining first spatiotemporal feature information provided in an embodiment of this application. Figure 9 This is a schematic diagram of eight time ranges provided for embodiments of this application.

[0183] Referring to the above description of S502, it can be determined that in this embodiment, the first time parameter and the first position parameter of the first image need to be encoded based on the above spatiotemporal coding rules (time coding rules and spatial coding rules) to obtain the first spatiotemporal feature information of the first image. Therefore, the process of obtaining the first spatiotemporal feature information of the first image will be described below.

[0184] 1. Obtain the first time parameters of the first image based on the information of the first image.

[0185] In this embodiment, the first time parameter may include at least one of the following: the season, holiday, and time range corresponding to the first image. Alternatively, the first image may not have a first time parameter.

[0186] In one implementation, the first time parameter of the first image can be determined based on the latitude and longitude of the first image and the time when the first image was captured.

[0187] We can first obtain the latitude and longitude of the first image and the shooting time, and then determine the first time parameter of the first image based on the latitude and longitude of the first image and the shooting time.

[0188] In one implementation, the latitude, longitude, and shooting time of an image can be determined based on its Exif information. Exif information is additional information embedded in the image file and is not part of the image's pixel data. The Exif information can be obtained using an Exif parser. In this embodiment, reference is made to... Figure 8 You can input an image into the image parsing module to output the image's capture time and latitude / longitude.

[0189] For example, the process by which the image parsing module processes a first image to output its capture time and latitude / longitude may include the following steps: The image parsing module first reads the input image file and identifies its format (e.g., JPEG, TIFF, etc.). Based on the image file format, the parsing module can locate the specific portion storing the Exif data. For example, for a JPEG file, the Exif data is typically located in the APP1 tag. The parsing module then parses each tag in the Exif data block, extracting the corresponding Exif information. After parsing, the parsing module can output the extracted Exif information to the user in an easy-to-understand and use format. This could be in text, list, or visualization format.

[0190] After obtaining the shooting time and latitude / longitude of the first image, the first time parameters of the first image, namely the season, holiday, and time range corresponding to the first image, can be determined based on the shooting time and latitude / longitude of the first image. In determining the time range corresponding to the first image based on its shooting time and latitude / longitude, the sunrise and sunset times of the day the first image was taken can first be determined. Then, the day can be divided into the eight time ranges mentioned above based on the sunrise and sunset times, and the time range corresponding to the first image can be determined based on the shooting time and the divided time ranges.

[0191] The process of determining the first time parameters of the first image based on its capture time and latitude / longitude can be referenced. Figure 8 To understand.

[0192] refer to Figure 8 The time and latitude / longitude of the first image are input into the season calculator, which then outputs the season information for the first image. Specifically, the season calculator can output the season corresponding to the first image. For example, the season calculator can achieve the functions described above by using the correspondence between time, latitude / longitude, and season.

[0193] refer to Figure 8 The time and latitude / longitude of the first image are input into the holiday calculator, which then outputs the holiday information associated with that image. Specifically, the holiday calculator can output the holiday corresponding to the first image; for example, it can achieve the functions described above by using the correspondence between time, latitude / longitude, and holidays.

[0194] refer to Figure 8The first image's capture time and latitude / longitude are input into the sunrise / sunset calculator. The calculator can then output the sunrise / sunset times corresponding to the first image. For example, the sunrise / sunset calculator can achieve the functions described above by using time, latitude / longitude, and the correspondence between sunrise and sunset. Based on these sunrise / sunset times, the day can then be divided into multiple time ranges.

[0195] In one implementation, the middle third of the time period from the previous day's sunset to today's sunrise can be defined as midnight, the time period from sunset plus half an hour to half an hour before sunrise can be defined as night, the time period from sunrise minus one hour can be defined as early morning, and the time period from sunrise plus or minus half an hour can be defined as sunrise.

[0196] Furthermore, the first third of the time period from sunrise to sunset is defined as morning, the middle third as noon, the last third as afternoon, and the period between sunrise and sunset plus or minus half an hour is defined as sunset. For example, the following combines... Figure 9 The rules for dividing the eight time ranges are explained in detail.

[0197] refer to Figure 9 Assuming the current sunrise time is 5:00 and sunset time is 17:00, according to the time range division rules mentioned above, the time range for midnight is 21:00 to 1:00, the time range for night is 17:30 to 3:59 and 00:00 to 4:30, the time range for early morning is 4:00 to 5:00, the time range for sunrise is 4:30 to 5:30, the time range for morning is 5:00 to 9:00, the time range for noon is 9:00 to 13:00, the time range for afternoon is 13:00 to 17:00, and the time range for sunset is 16:30 to 17:30.

[0198] Also refer to Figure 9 It can determine the encoding of eight time ranges: midnight, night, early morning, sunrise, morning, noon, afternoon, and sunset.

[0199] Finally, based on the shooting time and time range of the first image, the time range corresponding to the first image can be determined.

[0200] 2. Based on the information in the first image, obtain the first position parameters of the first image.

[0201] In this embodiment, the first location parameter may include at least one of the following: the geographical location corresponding to the capture of the first image, and the surrounding environment. Alternatively, the first image may not have a first location parameter. In one implementation, the first location parameter of the first image can be determined based on its latitude and longitude.

[0202] Based on the above introduction, it can be determined that the latitude and longitude of the first image can be determined according to its Exif information. Therefore, the process of obtaining the latitude and longitude of the first image will not be described in detail here.

[0203] refer to Figure 8 After obtaining the latitude and longitude of the image, it can be input into a spatial calculator to obtain the POI information of the first image. The POI information of the first image typically includes geographic location tags such as address, city name, and state name, as well as category tags such as "indoor location" and "outdoor attraction". This information may be directly contained in or implied in the name, description, or category of the POI.

[0204] It can also be understood that the latitude and longitude of an image can determine whether the image was taken in the Southern or Northern Hemisphere. Furthermore, based on the latitude and longitude, the time of capture, the city, country, and state, it is also possible to determine whether the image was taken in the south or north, and whether it is domestic or international.

[0205] 3. Based on the first time parameters of the first image, obtain the first time feature information of the first image.

[0206] You can refer to this. Figure 8 To understand this, after obtaining the first time parameter of the first image, encoding the first time parameter according to the first encoding rule yields the first time feature information of the first image. The first encoding rule is the time encoding rule described above. If the first image does not have a first time parameter, the first time feature information of the first image can be set to "0x00000".

[0207] For example, suppose there is an image that was taken at 5:10 AM on November 1st. Also suppose the sunrise time on that day was 5:00 AM and the sunset time was 5:00 PM. Then, we can determine that the first time parameter of this image is: winter, no holiday information, sunrise or morning. According to the first encoding rule, the first time feature information of this image can be determined as "0x40018". Here, "4" represents "winter", "00" indicates it is neither a solar nor a lunar holiday, and "18" (i.e., 00011000) indicates it belongs to both the sunrise and morning time ranges.

[0208] 4. Based on the first position parameters of the first image, obtain the first spatial feature information of the first image.

[0209] You can refer to this. Figure 8 To understand this, after obtaining the first positional parameter of the first image, encoding the first positional parameter according to the second encoding rule yields the first spatial feature information of the first image. The second encoding rule is the spatial encoding rule described above. If the first image does not have a first positional parameter, all bits of the first spatial feature information of the first image can be set to 0.

[0210] For example, suppose there exists an image containing the location parameters: Asia, Northern Hemisphere, North, Domestic, Indoor. According to the second encoding rule, the first spatial feature information of this image can be determined as "0001 00000000 10100101 0000". Here, "0001 0000 0000 101001" is the fourth field, and "010000" is the fifth field. For example, the encoding of the fourth field can be understood by referring to Table 2 below.

[0211] Table 2:

[0212] meaning state nation City Northern and Southern Hemispheres North and South Domestic and international Position parameters Asia / / Northern Hemisphere north domestic Field value 0001 0000 0000 10 10 01

[0213] Referring to Table 2 above, the field value for "State" is "0001," indicating that the location parameter of this image includes Asia. The field value for "Country" is "0000," indicating that this image does not contain information about any related countries. The field value for "City" is "0000," indicating that this image does not contain information about any related cities. The field value for "Northern and Southern Hemispheres" is "10," indicating that the location parameter of this image includes the Northern Hemisphere. The field value for "North and South" is "10," indicating that the location parameter of this image includes the North. The field value for "Domestic and International" is "01," indicating that the location parameter of this image includes China.

[0214] Furthermore, in the fifth field's code "010000", "01" represents "indoor" and "0000" means that the image does not contain information about the relevant attractions.

[0215] In this way, by determining the first time parameter and the first position parameter of the first image based on the Exif information, and encoding the first time parameter according to the first encoding rule and the first position parameter according to the second encoding rule, the spatiotemporal information contained in the first image can be effectively represented in numerical form, so as to facilitate subsequent operations.

[0216] Based on the above-introduced embodiments, the process of obtaining the second spatio-temporal feature information of the candidate text will be introduced in detail below.

[0217] As can be determined by referring to the introduction of S502 above, in this embodiment, based on the above spatio-temporal coding rules, the second time parameter and the second position parameter of the candidate text need to be encoded to obtain the second spatio-temporal feature information of the candidate text. Therefore, the process of obtaining the second spatio-temporal feature information of the candidate text will be described below.

[0218] 1. For any candidate text, according to the text features of the candidate text, obtain the second time parameter of the candidate text.

[0219] It can be understood that the processing method for each candidate text among multiple candidate texts in step 1 is similar. Therefore, below, any candidate text will be taken as an example for introduction.

[0220] In this embodiment, the second time parameter may include at least one of the following: the season, festival, and time range corresponding to the text content of the candidate text. At the same time, the candidate text may also have no second time parameter.

[0221] In one implementation, the second time parameter of the candidate text can be obtained by the keyword matching method. Specifically, some specific keywords can be set for each second time parameter, and then the content of the second time parameter can be determined based on the keywords matched by the candidate text. Exemplarily, for example, "early morning" and "morning" can be set as keywords for the time range of "morning". If the candidate text contains the keyword "early morning" or "morning", then this text may contain the time range of "morning". It can be understood that this method is applicable to the case where the number of candidate texts is small.

[0222] In another implementation, a language large model can be used to obtain the second time parameter of the candidate text. Specifically, the candidate text is input into the language large model, and the language large model can output the time information of the candidate text. Then, based on these time information, the second time parameter of the candidate text can be determined. Exemplarily, assume that a text "People step on the spring sun, happy Lantern Festival" is input into the language large model, and then the language large model can output time information such as "Lantern Festival", "daytime", "sunrise", etc. Then, based on these time information, it can be determined that this text may contain second time parameters such as "Lantern Festival", "morning", "noon", "afternoon", and "sunrise". It can be understood that this method is applicable to the case where the number of candidate texts is large.

[0223] 2. For any candidate text, according to the text features of the candidate text, obtain the second position parameter of the candidate text.

[0224] It is understandable that the processing method for each of the multiple candidate texts in step 2 is similar, so the following will take any candidate text as an example for introduction.

[0225] In this embodiment, the second location parameter may include at least one of the following: the geographical location corresponding to the text content of the candidate text, and the surrounding environment. Alternatively, the candidate text may not have a second location parameter.

[0226] The method for obtaining the second positional parameter of the candidate text is similar to the method for obtaining the second temporal parameter of the candidate text, so it will not be described in detail here.

[0227] 3. For any candidate text, encode the second time parameter according to the first encoding rule to obtain the second time feature information of the candidate text.

[0228] It is understandable that the processing method for each of the multiple candidate texts in step 3 is similar, so the following will take any candidate text as an example for introduction.

[0229] In this embodiment, the first encoding rule is the time encoding rule described above. After obtaining the second time parameter of a candidate text, the second time parameter can be encoded according to the first encoding rule to obtain the second time feature information of the candidate text. If the candidate text does not have a second time parameter, the second time feature information of the candidate text can be set to "0x00000".

[0230] For example, suppose there exists a text that reads "Work at sunrise, rest at sunset; day after day, thank you for accompanying me through every sunrise and sunset." Based on the first encoding rule mentioned above, the time feature information of this text can be determined to be "0x00088". Here, "000" indicates that there is no seasonal information or holiday information (i.e., it is neither a solar calendar holiday nor a lunar calendar holiday), and "88" (i.e., 10001000) indicates that it includes both sunrise and sunset time ranges.

[0231] It is important to understand that a text can describe a scene that occurs at a specific moment or within the eight specific time ranges mentioned above, or it can describe a scene that occurs within a general time range. This general time range can be a combination of multiple time ranges from the eight specific time ranges described above.

[0232] If a text describes a general time range, then when encoding the time range, multiple possible time ranges described by the text can be combined.

[0233] For example, suppose there is a text that reads "People stroll in the spring sunshine, joyful on the Lantern Festival." It can be understood that the scenario of "people stroll in the spring sunshine" could occur within four time ranges: morning, noon, afternoon, or sunrise. Therefore, the time range corresponding to the text can be morning, noon, afternoon, or sunrise. Thus, when encoding the time range of this text, these four time ranges can be combined to obtain the third field in the second time feature information of the text. This third field is "78" (a combination of "10", "20", "40", and "08").

[0234] 4. For any candidate text, encode the second position parameter according to the second encoding rule to obtain the second spatial feature information of the candidate text.

[0235] It is understandable that the processing method for each of the multiple candidate texts in step 4 is similar, so the following will take any candidate text as an example for introduction.

[0236] In this embodiment, the second encoding rule is the spatial encoding rule described above. After obtaining the second positional parameter of a candidate text, the second positional parameter can be encoded according to the first encoding rule to obtain the second spatial feature information of the candidate text. If the candidate text does not have a second positional parameter, all bits of the second spatial feature information of the candidate text can be set to 0.

[0237] In this way, by determining the second time parameter and the second position parameter of the candidate text based on keyword matching or a large semantic model, and by encoding the second time parameter according to the first encoding rule and the second position parameter according to the second encoding rule, the spatiotemporal information contained in the candidate text can be effectively represented in numerical form, so as to facilitate subsequent operations.

[0238] Based on the above-described embodiments, the following will further combine... Figure 10 The matching rules for the first spatiotemporal feature information of the first image and the second spatiotemporal feature information of any intermediate text are described in detail. Figure 10 This is a schematic diagram illustrating the process of filtering text based on a first condition set, as provided in an embodiment of this application.

[0239] Referring to the above description of S504, it can be determined that in this embodiment, it is necessary to match the first temporal feature information of the first image with the second temporal feature information of the intermediate text in order to filter out intermediate texts whose second spatiotemporal feature information does not match the first spatiotemporal feature information. Therefore, the implementation of the matching rule will be explained below.

[0240] It is understandable that when adding text descriptions to the first image, the scene or content presented in the first image can be described based on specific time information, such as seasons, holidays, and time periods within a day (such as morning, noon, and afternoon). Alternatively, the scene or content in the first image can be described without relying on time information.

[0241] Therefore, when matching the first image with the intermediate text, the intermediate text that has the same time information as the first image can be matched with the first image, and the intermediate text that does not contain time information can also be matched with the first image.

[0242] Based on the above approach, matching rules can be set for the temporal feature information of the first image and the intermediate text. Since the temporal feature information includes three fields: a first field indicating the season, a second field indicating the holiday, and a third field indicating the time range, it is necessary to discuss the matching rules for each field in the temporal feature information in detail. The matching rules for each field in the temporal feature information are described below, and these rules can be incorporated into the first condition set.

[0243] In this embodiment, when matching the first field, if the season indicated by the first field in the second time feature information is consistent with the season indicated by the first field in the first time feature information, or if the first field in the second time feature information does not indicate a season (i.e., the content of the intermediate text does not contain any description related to the season), then it can be determined that the first field in the second time feature information of the intermediate text matches the first field in the first time feature information of the first image. Otherwise, the intermediate text does not match the first image.

[0244] In one implementation, when matching the first field, if the first field in the second time feature information completely matches (i.e., the encoding is the same) the first field in the first time feature information, then it can be determined that the first field in the second time feature information of the intermediate text matches the first field in the first time feature information of the first image. It can be understood that during the matching process between the first field in the first time feature information and the first field in the second time feature information, as long as the first field in the second time feature information is consistent with the first field in the first time feature information, it means that the seasonal information reflected by the text is consistent with the seasonal information corresponding to the image. Therefore, it can be determined that the text and the image are a match.

[0245] Alternatively, in another special case, if the encoding of the first field in the second temporal feature information is "0", it means that the current text does not contain information corresponding to the season. It can be understood that using text without seasonal information as the text to be matched with the image will not actually result in a text-image mismatch. Therefore, when the encoding of the first field corresponding to the text is 0, regardless of the encoding of the first field corresponding to the image, it can be determined that the first field in the second temporal feature information of the intermediate text matches the first field in the first temporal feature information of the first image.

[0246] Apart from the two cases described above, it can be assumed that the intermediate text does not match the first image.

[0247] For example, suppose there is an image whose first temporal feature is "0x10000". Here, "0x10000" represents spring. Based on the matching rules described above, it can be determined that the first field "1" in the first temporal feature can match the second temporal feature where the first field is either "1" or "0". That is, this image can match the intermediate text where the second temporal feature is either "0x10000" or "0x00000".

[0248] In this embodiment, when matching the second field, if the festival indicated by the second field in the second time feature information is the same as the festival indicated by the second field in the first time feature information, or if the second field in the second time feature information does not indicate a festival (i.e., the content of the intermediate text does not contain any description related to a festival), then it can be determined that the second field in the second time feature information of the intermediate text matches the second field in the first time feature information of the first image. Otherwise, the intermediate text does not match the first image.

[0249] It should be understood that there may be cases where the festival indicated by the second field in the first time feature information includes both lunar and solar calendar holidays. In this case, if the festival indicated by the second field in the second time feature information is a lunar calendar holiday and / or a solar calendar holiday, then it can be determined that the second field in the second time feature information of the intermediate text matches the second field in the first time feature information of the first image. The festival indicated by the second field being a lunar calendar holiday and / or a solar calendar holiday can include the following three cases: the festival indicated by the second field is a lunar calendar holiday, the festival indicated by the second field is a solar calendar holiday, and the festival indicated by the second field includes both lunar and solar calendar holidays.

[0250] In one implementation, when matching the second field, if the Gregorian calendar position of the second field in the second time feature information completely matches the Gregorian calendar position of the second field in the first time feature information, then the Gregorian calendar position of the second field in the second time feature information of the intermediate text can match the Gregorian calendar position of the second field in the first time feature information of the first image. It can be understood that during the matching process between the second fields in the first and second time feature information, as long as the Gregorian calendar position of the second field in the second time feature information matches the Gregorian calendar position of the second field in the first time feature information, it means that the Gregorian calendar holiday information reflected in the text can be used to describe the Gregorian calendar holiday information corresponding to the image. Therefore, it can be determined that the text and image are matched.

[0251] At this point, a special case may exist: if the Gregorian calendar bit encoding of the second field in the second time feature information is "0", it means that the current text does not contain information corresponding to Gregorian calendar holidays. It can be understood that using text that does not contain Gregorian calendar holiday information as the text to be matched with the image will not actually result in a text-image mismatch. Therefore, when the Gregorian calendar bit encoding of the second field corresponding to the text is 0, regardless of the Gregorian calendar bit encoding of the second field corresponding to the image, it can be determined that the Gregorian calendar bit of the second field in the second time feature information of the intermediate text matches the Gregorian calendar bit of the second field in the first time feature information of the first image.

[0252] Alternatively, in another scenario, if the lunar calendar position of the second field in the second time feature information completely matches the lunar calendar position of the second field in the first time feature information, then the lunar calendar position of the second field in the second time feature information of the intermediate text can match the lunar calendar position of the second field in the first time feature information of the first image. It can be understood that during the matching process between the second fields in the first and second time feature information, as long as the lunar calendar position of the second field in the second time feature information matches the lunar calendar position of the second field in the first time feature information, it means that the lunar festival information reflected in the text can be used to describe the lunar festival information corresponding to the image. Therefore, it can be determined that the text and image are matched.

[0253] At this point, a special case may exist: if the lunar calendar digit encoding of the second field in the second time feature information is "0", it means that the current text does not contain information corresponding to lunar festivals. It can be understood that using text that does not contain lunar festival information as the text to be matched with the image will not actually result in a text-image mismatch. Therefore, when the lunar calendar digit encoding of the second field corresponding to the text is 0, regardless of the lunar calendar digit encoding of the second field corresponding to the image, it can be determined that the lunar calendar digit of the second field in the second time feature information of the intermediate text matches the lunar calendar digit of the second field in the first time feature information of the first image.

[0254] Apart from the four situations described above, it can be assumed that the intermediate text does not match the first image.

[0255] For example, suppose there exists an image whose first time signature is "0x02200". Here, "0x02000" represents Valentine's Day, and "0x00200" represents the Lantern Festival. Based on the matching rules described above, it can be determined that the second field "22" in the first time signature can match second time signatures where the second field is "22", "20", "02", or "00". Therefore, this image can match the middle text where the second time signature is "0x02200", "0x00200", "0x02000", or "0x00000".

[0256] Furthermore, it can be understood that an image can depict a scene occurring at a specific moment, while a piece of text can depict a scene occurring at a specific moment or within the aforementioned eight specific time ranges, or it can depict a scene occurring within a general time range. This general time range can be multiple time ranges from the aforementioned eight specific time ranges. For example, the text "People stroll in the spring sunshine, enjoying the Lantern Festival" described above.

[0257] In this situation, it's possible that an image corresponds to one time range, while a text corresponds to multiple time ranges, and the time range corresponding to the text includes the time range corresponding to the image. It's understandable that the text description is more general. Therefore, it can be determined that the image matches the text.

[0258] Based on the above approach, matching rules can be set for the third field of time feature information. In this embodiment, when matching the third field, if the first time range indicated by the third field in the second time feature information (the time range corresponding to the intermediate text) overlaps with the second time range indicated by the third field in the first time feature information (the time range corresponding to the first image), or if the third field in the second time feature information does not indicate a time range (i.e., the content of the intermediate text does not contain any description related to the time range), then the third field in the second time feature information of the intermediate text can match the third field in the first time feature information of the first image. Otherwise, the intermediate text and the first image do not match.

[0259] In one implementation, when matching the third field, a bitwise AND operation is performed between the third field in the second time feature information and the third field in the first time feature information. If the result is not "0", then the third field in the second time feature information of the intermediate text can match the third field in the first time feature information of the first image. It can be understood that during the matching process between the third field in the first time feature information and the third field in the second time feature information, as long as there is a bit with a result of 1 after the AND operation, it means that the time period information reflected by the text covers the time period information corresponding to the image. Therefore, the bitwise AND operation logic can be used to determine whether the text and image match.

[0260] Alternatively, in another special case, if the code for the third field in the second time feature information is "00", it means that the current text does not contain information corresponding to the time range. It can be understood that using text without time period information as the text to be matched with the image will not actually result in a text-image mismatch. Therefore, when the code for the third field corresponding to the text is 00, regardless of the code for the third field corresponding to the image, it can be determined that the third field in the second time feature information of the intermediate text matches the third field in the first time feature information of the first image.

[0261] Apart from the two cases described above, it can be assumed that the intermediate text does not match the first image.

[0262] For example, suppose there is an image whose first time feature information is "0x00018". Here, "0x00010" indicates morning, and "0x00008" indicates sunrise. Based on the matching rules described above, it can be determined that the third field "18" (00011000) in the first time feature information can match second time feature information where the third field is "18" (00011000), "10" (00010000), "08" (00001000), or "00" (000000000). Furthermore, the third field "18" in the first time feature information can also match second time feature information where the third field is a combination of time periods such as "78" (01111000, "10", "20", "40", and "08"). Then this image can be matched with the middle text whose second time feature information is “0x00018”, “0x00010”, “0x00008”, “0x00000” or “0x00078”.

[0263] It can be understood that, based on the aforementioned first set of conditions, the first temporal feature information of the image can be matched with the second temporal feature information of multiple intermediate texts. If the second temporal feature information of the intermediate text does not satisfy any condition in the first set of conditions, then the intermediate text is filtered out. The following section combines... Figure 10 Let me explain the process of filtering text based on the first set of conditions.

[0264] refer to Figure 10 Suppose there exists an image whose first-time feature information is "0x42218", and six texts: text 1, text 2, text 3, text 4, text 5, and text 6.

[0265] Text 1 reads "Lantern Festival, may all your dreams come true," with a second time signature of "0x00200"; Text 2 reads "People stroll in the spring sunshine, joyful Lantern Festival," with a second time signature of "0x00278"; Text 3 reads "The sun rises in the east, shining brightly," with a second time signature of "0x00008"; Text 4 reads "Enjoy the moon at Mid-Autumn Festival, talk about reunion," with a second time signature of "0x00503"; Text 5 reads "Lights sway, a unique beauty on the Lantern Festival night," with a second time signature of "0x00203"; and Text 6 reads "Day after day, thank you for accompanying us every sunrise and sunset," with a second time signature of "0x00088."

[0266] refer to Figure 10 It can be determined that "0x40000" represents winter, "0x02200" represents Valentine's Day and the Lantern Festival, and "0x00018" represents "morning" and "sunrise".

[0267] Based on the above matching rules, it can be determined that "0x40000" can match "0x40000" and "0x00000". "0x02200" can match "0x02200", "0x02000", "0x00200", and "0x00000". "0x00018" can match "0x00018", "0x00010", "0x00008", "0x00000", and "0x00078".

[0268] Then “0x42218” can match “0x002000”, “0x00278”, “0x00008” and “0x00088”, but “0x42218” does not match “0x00503” (“5” does not represent the Lantern Festival and “03” represents midnight and night) and “0x00203” (“03” represents midnight and night).

[0269] Therefore, this image can be matched with text 1, text 2, text 3, and text 6, but not with text 4 and text 5.

[0270] Next, we will introduce the matching rules between the first spatial feature information of the first image and the second spatial feature information of any intermediate text:

[0271] It is understandable that when adding text descriptions to the first image, the scene or content presented in the first image can be described based on specific spatial information, such as the state, country, city, and environment. Alternatively, the scene or content in the first image can be described without relying on spatial information.

[0272] Therefore, when matching the first image with the intermediate text, the intermediate text that has the same spatial information as the first image can be matched with the first image, and the intermediate text that does not contain spatial information description can also be matched with the first image.

[0273] Based on the above approach, matching rules can be set for the spatial feature information of the first image and the intermediate text. Since the spatial feature information includes a fourth field indicating geographical location and a fifth field indicating the surrounding environment, it is necessary to discuss the matching rules for each field in the spatial feature information in detail. The matching rules for each field in the spatial feature information are described below, and these rules can be incorporated into the second condition set.

[0274] In this embodiment, when matching the fourth field, if the geographical location indicated by the fourth field in the second spatial feature information is consistent with the geographical location indicated by the fourth field in the first spatial feature information, or if the fourth field in the second spatial feature information does not indicate a geographical location (i.e., the content of the intermediate text does not contain any description related to a geographical location), then it can be determined that the fourth field in the second spatial feature information of the intermediate text matches the fourth field in the first spatial feature information of the first image. Otherwise, the intermediate text does not match the first image.

[0275] When matching the fourth field, if the environment indicated by the fifth field in the second spatial feature information is consistent with the environment indicated by the fifth field in the first spatial feature information, or if the fifth field in the second spatial feature information does not indicate the environment (i.e., the content of the intermediate text does not contain a description related to the environment), then it can be determined that the fifth field in the second spatial feature information of the intermediate text matches the fifth field in the first spatial feature information of the first image. Otherwise, the intermediate text does not match the first image.

[0276] It can be understood that, based on the aforementioned second set of conditions, the first spatial feature information of the image can be matched with the second spatial feature information of multiple intermediate texts. If the second spatial feature information of an intermediate text does not satisfy any item in the second set of conditions, then that intermediate text is filtered out.

[0277] In this way, the matching method can either match text containing explicit spatiotemporal information to the first image, which is highly targeted, or match text without spatiotemporal information to the first image, which is more general, thus increasing the diversity of the accompanying text.

[0278] The following is combined Figure 11 The image processing methods will be introduced in detail. Figure 11 This is a signaling interaction diagram of the image processing method provided in the embodiments of this application.

[0279] 1. The text management module sends the candidate text to the semantic parsing module.

[0280] The text management module can have a text library containing multiple candidate texts. Before semantically matching the candidate texts with the first image, it's necessary to obtain the second semantic feature information for each candidate text. Therefore, the text management module sends multiple candidate texts to the semantic parsing module to obtain their second semantic feature information.

[0281] 2. After receiving the candidate text, the semantic parsing module sends a request to the semantic model to obtain the semantic meaning.

[0282] After the semantic parsing module receives the candidate text, it can send a request to the semantic model to obtain the semantics and send the candidate text to the semantic model to obtain the semantics of the candidate text.

[0283] 3. The semantic model returns the semantics of the candidate text to the semantic parsing module.

[0284] In this embodiment, the semantic model is a pre-trained semantic model. When the candidate text is input into the semantic model, the semantic model can output the semantics of the candidate text and return it to the semantic parsing module.

[0285] 4. The semantic parsing module returns the second semantic feature information to the text management module.

[0286] After obtaining the semantics of the candidate text, the semantic parsing module can process it to obtain the second semantic feature information of the text and return it to the text management module.

[0287] For the specific implementation of steps 1, 2, 3 and 4, please refer to the content of step S503.

[0288] 5. The text management module sends the text to be selected to the text encoding module.

[0289] In addition to semantic matching between multiple candidate texts and the first image, coded matching between the candidate texts and the first image is also required. The encoding process for the candidate texts can be implemented by a text encoding module. Therefore, the candidate texts are input into the text encoding module to obtain the second spatiotemporal feature information.

[0290] 6. The text encoding module returns the second spatiotemporal feature information to the text management module.

[0291] After obtaining the candidate text, the text encoding module can process it. For example, it can obtain a second time parameter and a second position parameter from the candidate text, and encode these parameters to obtain second spatiotemporal feature information. Then, the second spatiotemporal feature information is returned to the text management module.

[0292] For the specific implementation of steps 5 and 6, please refer to the content in step S502.

[0293] 7. When the first image is input to the image management module.

[0294] When it is necessary to add text to the first image, semantic matching, encoding matching, and text selection can be performed on the first image and the candidate text to obtain the target text for the first image. At this time, the first image can be input into the image management module to obtain the target text for the first image.

[0295] 8. The image management module sends the first image to the image parsing module.

[0296] When the image management module receives the first image, it can send the first image to the image parsing module to obtain the first spatiotemporal feature information of the image.

[0297] 9. The image parsing module performs Exif parsing on the first image.

[0298] 10. The image parsing module sends spatiotemporal information to the image encoding module.

[0299] When the image parsing module receives the first image, it can first obtain the Exif information of the first image and parse the Exif information to obtain the spatiotemporal information of the first image. The spatiotemporal information here can refer to the first time parameter and the first position parameter. After obtaining the first time parameter and the first position parameter of the first image, the first spatiotemporal parameter can be input to the image encoding module to obtain the first spatiotemporal feature information of the first image.

[0300] 11. The image encoding module returns the first spatiotemporal feature information to the image parsing module.

[0301] After the image encoding module receives the first time parameter and the first position parameter of the first image, it can obtain the first spatiotemporal feature information of the image according to the first encoding rule and the second encoding rule mentioned above, and return the first spatiotemporal feature information to the image parsing module. The specific implementation of obtaining the first spatiotemporal feature information can be found in step S502.

[0302] 12. The image parsing module returns the first spatiotemporal feature information to the image management module.

[0303] For the specific implementation of steps 8, 9, 10, 11 and 12, please refer to the content in step S502.

[0304] 13. The image management module sends the first image to the semantic parsing module.

[0305] In addition to obtaining the first spatiotemporal feature information of the first image, it is also necessary to obtain the first semantic feature information of the first image. This process can be implemented by the semantic parsing module and the semantic model. Therefore, the first image is sequentially input into the semantic parsing module to obtain the first semantic feature information of the first image.

[0306] 14. The semantic parsing module receives the first image and sends a request to the semantic model to obtain semantics.

[0307] 15. The semantic model returns the semantics of the first image.

[0308] 16. The semantic parsing module returns the first semantic feature information to the image management module.

[0309] Steps 14, 15, and 16 are similar to steps 2, 3, and 4 above, so they will not be repeated here.

[0310] 17. The image management module sends a caption request, along with the first semantic feature information and the first spatiotemporal feature information, to the image captioning module.

[0311] After obtaining the first semantic feature information and the first spatiotemporal feature information of the first image, the image management module can send a request to the image captioning module to obtain the target text, and send the first semantic feature information and the first spatiotemporal feature information of the first image to it.

[0312] 18. After receiving the request to obtain the target text, the image captioning module sends a request to the text management module to obtain the second semantic feature information and the second spatiotemporal feature information.

[0313] 19. After receiving the request to obtain the second semantic feature information and the second spatiotemporal feature information, the text management module returns the second semantic feature information and the second spatiotemporal feature information to the image captioning module.

[0314] 20. The image-text matching module performs semantic matching.

[0315] After obtaining the second semantic feature information and second spatiotemporal feature information of each of the multiple candidate texts, the image captioning module performs semantic matching processing on the first semantic feature information of the first image and the second semantic feature information of the multiple candidate texts to filter out a preset number of candidate texts with a high degree of matching as intermediate texts. The specific implementation of this step can be found in step S503.

[0316] 21. The image and text matching module performs encoding matching.

[0317] After obtaining the intermediate text, the second spatiotemporal feature information of the intermediate text and the first spatiotemporal feature information of the first image can be encoded and matched to filter out intermediate text that does not match the first image. The specific implementation of this step can be found in step S504.

[0318] 22. The image-text matching module optimizes text selection.

[0319] After obtaining the filtered intermediate text, the image captioning module can randomly select one text as the target text, or select an intermediate text as the target text according to specific rules. For details on the implementation of this step, please refer to step S505.

[0320] 23. After the image captioning module obtains the target text of the first image, the image captioning module returns the target text to the image management module.

[0321] It should be noted that the module names involved in the embodiments of this application can all be defined as other names, as long as they can achieve the function of each module, and no specific restrictions are placed on the module names.

[0322] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0323] The image processing method of the present application embodiments has been described above. The apparatus for performing the above method provided in the present application embodiments is described below. Those skilled in the art will understand that the methods and apparatus can be combined with and referenced by each other, and the related apparatus provided in the present application embodiments can perform the steps in the above image processing method.

[0324] The image processing method provided in this application can be applied to electronic devices with image processing capabilities. Electronic devices include terminal devices, and the specific device form of the terminal device can be referred to the above-described related information, which will not be repeated here.

[0325] In one implementation, this application provides an electronic device. Figure 12 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application.

[0326] like Figure 12 As shown, the electronic device 1200 includes: a processor 1201 and a memory 1202; the memory 1202 stores computer execution instructions; the processor 1201 executes the computer execution instructions stored in the memory 1202, causing the electronic device 1200 to perform the above-described method.

[0327] When the memory 1202 is set up independently, the electronic device also includes a bus 1203 for connecting the memory 1202 and the processor 1201.

[0328] This application provides a chip. The chip includes a processor, which calls a computer program in memory to execute the technical solutions in the above embodiments. Its implementation principle and technical effects are similar to those in the related embodiments described above, and will not be repeated here.

[0329] This application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, it implements the methods described above. The methods described in the above embodiments can be implemented wholly or partially by software, hardware, firmware, or any combination thereof. If implemented in software, the functionality can be stored as one or more instructions or code on or transmitted over the computer-readable medium. The computer-readable medium can include computer storage media and communication media, and can also include any medium that can transfer a computer program from one place to another. The storage medium can be any target medium accessible by a computer.

[0330] In one possible implementation, a computer-readable medium may include RAM, ROM, compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage or other magnetic storage devices, or any other medium targeted to carry or to store the required program code in the form of instructions or data structures, and accessible by a computer. Furthermore, any connection is appropriately referred to as a computer-readable medium. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. As used herein, disks and optical discs include optical discs, laser discs, optical discs, Digital Versatile Discs (DVDs), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, while optical discs optically reproduce data using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0331] This application provides a computer program product, which includes a computer program that, when run, causes a computer to perform the above-described method.

[0332] This application describes embodiments of methods, apparatus (systems), and computer program products according to embodiments of this application with reference to flowchart illustrations and / or block diagrams. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, special-purpose computer, embedded processor, or other programmable device to produce a machine, such that the instructions, which execute via the processing unit of the computer or other programmable data processing device, generate instructions for implementing the flowchart illustrations. Figure 1One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0333] The above specific embodiments further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.

Claims

1. An image processing method, characterized in that, include: Matching is performed based on the first semantic feature information of the first image and the second semantic feature information of each of the multiple candidate texts, so as to select multiple intermediate texts from the multiple candidate texts; Matching is performed based on the first spatiotemporal feature information of the first image and the second spatiotemporal feature information of each of the plurality of intermediate texts to determine the target text among the plurality of intermediate texts; The target text is displayed for the first image.

2. The method according to claim 1, characterized in that, The step of matching the first spatiotemporal feature information of the first image and the second spatiotemporal feature information of each of the plurality of intermediate texts to determine the target text among the plurality of intermediate texts includes: Among the plurality of intermediate texts, intermediate texts whose second spatiotemporal feature information does not match the first spatiotemporal feature information are filtered out; The target text is determined from the remaining intermediate text after filtering.

3. The method according to claim 2, characterized in that, The first spatiotemporal feature information includes first temporal feature information and first spatial feature information, and the second spatiotemporal feature information includes second temporal feature information and second spatial feature information; The step of filtering out intermediate texts from the plurality of intermediate texts where the second spatiotemporal feature information does not match the first spatiotemporal feature information includes: Among the plurality of intermediate texts, intermediate texts whose second temporal feature information does not match the first temporal feature information are filtered out, and intermediate texts whose second spatial feature information does not match the first spatial feature information are also filtered out.

4. The method according to claim 3, characterized in that, The time feature information includes an encoding consisting of M fields, where M is an integer greater than or equal to 1; The M fields include at least one of the following: a first field for indicating a season, at least one second field for indicating a festival, and a third field for indicating a time range.

5. The method according to claim 4, characterized in that, The step of filtering out intermediate texts from the plurality of intermediate texts where the second time feature information does not match the first time feature information includes: For any of the intermediate texts, if the second time feature information of the intermediate text does not satisfy any item in the first condition set, then the intermediate text is filtered. The first set of conditions includes: The first field in the second time feature information is consistent with the season indicated by the first field in the first time feature information, or the first field in the second time feature information does not indicate a season; The second field in the second time feature information is consistent with the festival indicated by the second field in the first time feature information, or the second field in the second time feature information does not indicate a festival; The first time range indicated by the third field in the second time feature information overlaps with the second time range indicated by the third field in the first time feature information, or the third field in the second time feature information does not indicate a time range.

6. The method according to claim 3, characterized in that, The spatial feature information includes an encoding consisting of N fields, where N is an integer greater than or equal to 1; The N fields include at least one of the following: at least one fourth field indicating the geographical location, and a fifth field indicating the environment, wherein the geographical location indicated by the fourth field corresponds to a preset geographical level.

7. The method according to claim 6, characterized in that, The step of filtering out intermediate texts from the plurality of intermediate texts where the second spatial feature information does not match the first spatial feature information includes: For any one of the intermediate texts, if the second spatial feature information of the intermediate text does not satisfy any item in the second condition set, then the intermediate text is filtered. The second set of conditions includes: The fourth field in the second spatial feature information is consistent with the geographical location indicated by the fourth field in the first spatial feature information, or the fourth field in the second temporal feature information does not indicate a geographical location; The fifth field in the second spatial feature information is consistent with the environment indicated by the fifth field in the first spatial feature information, or the fifth field in the second temporal feature information does not indicate the environment.

8. The method according to any one of claims 3-7, characterized in that, The method further includes: A first time parameter of the first image is obtained, and the first time parameter is encoded according to a first encoding rule to obtain the first time feature information of the first image; wherein, the first time parameter includes at least one of the following: the season, festival, and time range corresponding to the capture of the first image; and, For any of the candidate texts, a second time parameter of the candidate text is obtained, and the second time parameter is encoded according to the first encoding rule to obtain the second time feature information of the candidate text; wherein, the second time parameter includes at least one of the following: the season, festival and time range corresponding to the text content of the candidate text.

9. The method according to any one of claims 3-7, characterized in that, The method further includes: A first location parameter of the first image is obtained, and the first location parameter is encoded according to a second encoding rule to obtain the first spatial feature information of the first image; wherein, the first location parameter includes at least one of the following: the geographical location corresponding to the capture of the first image, and the surrounding environment; and For any of the candidate texts, a second position parameter of the candidate text is obtained, and the second position parameter is encoded according to the second encoding rule to obtain the second spatial feature information of the candidate text; wherein, the second position parameter includes at least one of the following: the geographical location corresponding to the text content of the candidate text, and the environment in which it is located.

10. The method according to claim 8 or 9, characterized in that, The step of obtaining the second time parameter of the candidate text includes: Based on the text features of the candidate text, obtain the second time parameter of the candidate text; The step of obtaining the second position parameter of the candidate text includes: Based on the text features of the candidate text, obtain the second positional parameter of the candidate text.

11. An electronic device, characterized in that, The electronic device includes: one or more processors and memory; The memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, the one or more processors invoking the computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 10.

12. A chip system, characterized in that, The chip system is applied to an electronic device, the chip system including one or more processors, the one or more processors being used to invoke computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1 to 10.

14. A computer program product, characterized in that, The computer program product includes computer program code that, when run on an electronic device, causes the electronic device to perform the method as described in any one of claims 1 to 10.

Images