Object processing method and device and electronic equipment

Abstract

The invention provides an object processing method, which comprises the following steps: processing a to-be-processed object to obtain key feature information and initial feature information of the to-be-processed object, the initial feature information comprising local feature information and global feature information; performing fusion processing on the key feature information, the local feature information and the global feature information to obtain target feature information of the to-be-processed object, the key feature information being target frame information or text feature information based on the to-be-processed object of different modalities; and determining that an object meeting a similarity condition with the target feature information of the to-be-processed object is a result object corresponding to the to-be-processed object, the modality of the to-be-processed object is different from the modality of the result object, and the modality comprises at least one of a text, an image, a video and voice. The invention further provides an object processing device and electronic equipment.

Description

Technical Field

[0001] The present disclosure relates to the field of cross-modal retrieval technology, and in particular to an object processing method, device and electronic device. Background Art

[0002] The retrieval technology that uses multimodal models to achieve retrieval between different modal information (such as text to image or image to text) is usually called cross-modal retrieval. Through a multimodal deep learning model, information of different modalities is mapped to a shared representation space, so that these two different types of information can be effectively compared and matched in the same space. However, the features currently extracted from text and image information using multimodal models are limited to global features, and different modal information is processed in the same way, which easily leads to the neglect of important details, making the results of cross-modal retrieval less accurate and difficult to meet the actual needs of users. Summary of the invention

[0003] One aspect of the present disclosure provides an object processing method, including: processing an object to be processed to obtain key feature information and initial feature information of the object to be processed, wherein the initial feature information includes local feature information and global feature information; fusing the key feature information, the local feature information and the global feature information to obtain target feature information of the object to be processed, wherein the key feature information is target frame information or text feature information of the object to be processed based on different modalities; and determining that an object that meets a similarity condition with the target feature information of the object to be processed is a result object corresponding to the object to be processed, wherein the modality of the object to be processed is different from that of the result object, and the modality includes at least one of text, image, video and voice.

[0004] According to an embodiment of the present disclosure, target feature information is obtained based on key feature information, local feature information and global feature information, including: processing local feature information based on key feature information to obtain fine-grained feature information; and fusing the fine-grained feature information and global feature information to obtain target feature information.

[0005] According to an embodiment of the present disclosure, when the object to be processed is an image or a video, the key feature information is the target frame information, and the target frame information is obtained by detecting the object to be processed; the local feature information is processed based on the key feature information to obtain fine-grained feature information, including: determining the fine-grained feature information based on overlapping image information between the local feature information and the target frame information.

[0006] According to an embodiment of the present disclosure, when the object to be processed is speech or text, the key feature information is text feature information; local feature information is processed based on the key feature information to obtain fine-grained feature information, including: determining the fine-grained feature information based on overlapping text information between the local feature information and the text feature information.

[0007] According to an embodiment of the present disclosure, the method also includes: performing word segmentation processing on speech or text to obtain multiple word segmentation features; and determining the word segmentation features that are the same as the text information corresponding to the candidate image from the multiple word segmentation features as text feature information, and the candidate image is obtained based on text or speech retrieval.

[0008] According to an embodiment of the present disclosure, fine-grained feature information includes fine-grained features and a first weight corresponding to the fine-grained features; the fine-grained feature information and the global feature information are fused to obtain target feature information, including: updating the initial weight of the global feature information based on the first weight to obtain a second weight corresponding to the global feature information, and the sum of the first weight and the second weight satisfies the target weight value; and weighted fusion of the fine-grained features and the global feature information based on the first weight and the second weight, respectively, to determine the target feature information.

[0009] According to an embodiment of the present disclosure, the object processing method is executed by a deep learning model, and the deep learning model includes an information detection layer and an information recognition layer; the training process of the deep learning model includes: inputting the first sample object to be processed into the information detection layer in the deep learning model to obtain sample target frame information; and inputting the first sample object to be processed and the second sample object to be processed into the information recognition layer to obtain the first sample local feature information, the first sample global feature information and the second sample local feature information, and the second sample global feature information; the modality of the first sample object to be processed is different from the modality of the second sample object to be processed.

[0010] According to an embodiment of the present disclosure, a deep learning model is trained by a loss function determined by the following operations: determining first sample key feature information and second sample key feature information corresponding to a first sample object to be processed and a second sample object to be processed respectively based on sample target frame information; determining sample target image feature information and sample target text feature information based on the first sample key feature information, the second sample key feature information, the first sample local feature information, the first sample global feature information, the second sample local feature information and the second sample global feature information to obtain sample feature pairs; determining a loss function based on the similarity between sample feature pairs and label similarity; label similarity is obtained based at least on the original similarity between the first sample object to be processed and the second sample object to be processed.

[0011] Another aspect of the present disclosure provides an object processing device, including: an information output module, used to process an object to be processed, and obtain key feature information and initial feature information of the object to be processed, wherein the initial feature information includes local feature information and global feature information; a target feature information acquisition module, used to fuse the key feature information, the local feature information and the global feature information, and obtain target feature information of the object to be processed, wherein the key feature information is target frame information or text feature information of the object to be processed based on different modalities; and a result generation module, used to determine that an object that meets a similarity condition with the target feature information of the object to be processed is a result object corresponding to the object to be processed, wherein the modality of the object to be processed is different from that of the result object, and the modality includes at least one of text, image, video and voice.

[0012] Another aspect of the present disclosure provides an electronic device, comprising: one or more processors; a memory for storing one or more computer programs, characterized in that the one or more processors execute the one or more computer programs to implement the steps of the above-mentioned object processing method.

[0013] Another aspect of the present disclosure provides a non-volatile storage medium storing computer executable instructions, which are used to implement any of the above methods when executed.

[0014] Another aspect of the present disclosure provides a computer program, which includes computer executable instructions, and the instructions are used to implement any of the above methods when executed. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] For a more complete understanding of the present disclosure and its advantages, reference will now be made to the following description taken in conjunction with the accompanying drawings, in which:

[0016] Figure 1 The application scenarios of the object processing method, device and electronic device according to the embodiments of the present disclosure are shown;

[0017] Figure 2 A flowchart of an object processing method according to an embodiment of the present disclosure is schematically shown;

[0018] Figure 3A An example diagram schematically shows a process for determining fine-grained fusion feature information of an image according to an embodiment of the present disclosure;

[0019] Figure 3B An example diagram schematically shows a process for determining text coarse-grained and fine-grained fusion feature information according to an embodiment of the present disclosure;

[0020] Figure 3C An example diagram of the training process of a deep learning model according to an embodiment of the present disclosure is schematically shown;

[0021] Figure 4 A block diagram of an object processing device according to an embodiment of the present disclosure is shown;

[0022] Figure 5 A schematic block diagram of an example electronic device that can be used to implement the object processing method according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0023] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0024] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0025] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0026] Some block diagrams and / or flow charts are shown in the accompanying drawings. It should be understood that some blocks or combinations thereof in the block diagrams and / or flow charts may be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that these instructions, when executed by the processor, may create a device for implementing the functions / operations described in these block diagrams and / or flow charts.

[0027] Therefore, the technology of the present disclosure can be implemented in the form of hardware and / or software (including firmware, microcode, etc.). In addition, the technology of the present disclosure can take the form of a computer program product on a computer-readable medium storing instructions, which can be used by an instruction execution system or in combination with an instruction execution system. In the context of the present disclosure, a computer-readable medium can be any medium that can contain, store, transmit, propagate, or transmit instructions. For example, a computer-readable medium can include, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, device, or propagation medium. Specific examples of computer-readable media include: magnetic storage devices, such as magnetic tape or hard disk (HDD); optical storage devices, such as compact disk (CD-ROM); memory, such as random access memory (RAM) or flash memory; and / or wired / wireless communication links.

[0028] Figure 1 The application scenarios of the object processing method, device and electronic device according to the embodiments of the present disclosure are shown.

[0029] It should be noted that Figure 1 The examples shown are merely scenarios in which the embodiments of the present disclosure can be applied, to help those skilled in the art understand the technical content of the present disclosure, but do not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0030] like Figure 1 As shown, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0031] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103.

[0032] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0033] The server 105 may be a server that provides various services. For example, the object to be processed may be processed in the server 105 to obtain key feature information and initial feature information. For another example, the key feature information, local feature information, and global feature information may be fused in the server 105 to obtain target feature information, thereby determining a processing result corresponding to the object to be processed, and feeding back the processing result (result object) to the terminal device.

[0034] It should be noted that the object processing method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the object processing device provided in the embodiment of the present disclosure can generally be set in the server 105. The object processing method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the object processing device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0035] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0036] In some examples, multimodal models (such as Contrastive Language-Image Pretraining, CLIP) are used in cross-modal retrieval tasks to extract image features and their corresponding image features for contrastive learning. Image features and text features are represented by a one-dimensional vector respectively, and paired image and text vectors are aligned in space to complete image classification, text search image, and image search text tasks.

[0037] However, the above method simply extracts a global feature from the image and text for calculation. During the feature extraction process, all image blocks and text words are treated the same, resulting in the extracted features being more inclined to global features, making the retrieval results less accurate.

[0038] In other examples, attempts are made to determine which image parts and which text parts are most important for the task by introducing an attention mechanism into the multimodal model architecture, assigning weights to each image patch and each word.

[0039] However, for tasks like zero-shot learning, the model has not received task-specific data and cannot "reinforce" the model's understanding of certain tasks through a large number of task-related examples. Simple implicit weight learning is not enough to provide sufficient knowledge, causing the extracted features to ignore the details in the image, which in turn leads to decreased retrieval efficiency and accuracy in the task of text retrieval of images, reducing the user experience.

[0040] Based on the above problems, the present disclosure provides an object processing method, including processing an object to be processed to obtain key feature information and initial feature information of the object to be processed, the initial feature information including local feature information and global feature information; fusing the key feature information, the local feature information and the global feature information to obtain target feature information of the object to be processed, the key feature information being target frame information or text feature information of the object to be processed based on different modalities; and determining that an object that meets a similarity condition with the target feature information of the object to be processed is a result object corresponding to the object to be processed, the modality of the object to be processed is different from that of the result object, and the modality includes at least one of text, image, video and voice.

[0041] According to the embodiments of the present disclosure, by fusing the key feature information, local feature information and global feature information of the object to be processed, the target feature information containing coarse and fine granularity fusion is obtained, thereby determining the result object corresponding to the object to be processed. Since the target feature information is determined based on the key feature information (target frame information or text feature information), on the basis of retaining the original global features, the extracted target frame information or text feature information is added with fine-grained features, which enhances the perception of fine-grained image and text information in cross-modal retrieval, improves the representation ability of the entire feature, and further enhances the user experience.

[0042] Figure 2 The flowchart of the object processing method according to the embodiment of the present disclosure is schematically shown.

[0043] Specifically, Figure 2 As shown, the method includes operations S201 to S203.

[0044] In operation S201, the object to be processed is processed to obtain key feature information and initial feature information of the object to be processed, where the initial feature information includes local feature information and global feature information.

[0045] In the embodiments of the present disclosure, the objects to be processed may be objects of different modalities, including but not limited to images, text, time-frequency, and audio, and are used in scenarios such as image search engines, image annotation, visual question answering, and multimodal translation. The key feature information can be divided into target frame information and text feature information according to the different modalities of the objects to be processed, and obtained through the information detection layer in the multimodal model. Local feature information and global feature information can be obtained through the information recognition layer in the multimodal model.

[0046] For example, the information detection layer in the multimodal model is used to detect the image to obtain the target frame information of the image; the information recognition layer in the multimodal model is used to recognize the image to extract local feature information and global feature information.

[0047] For example, the input text is segmented and retrieved to obtain segmentation features and a set of candidate images corresponding to the text, and the candidate images are detected using the information detection layer in the multimodal model to obtain text information, thereby determining from the segmentation features that the features that are the same as the text information as text feature information; the text is recognized using the information recognition layer in the multimodal model to extract local feature information and global feature information corresponding to the text.

[0048] In operation S202, the key feature information, the local feature information and the global feature information are fused to obtain target feature information of the object to be processed. The key feature information is target frame information or text feature information of the object to be processed based on different modalities.

[0049] In the embodiments of the present disclosure, local feature information based on different modalities of the objects to be processed may include text local feature information or image local feature information; global feature information based on different modalities of the objects to be processed may include text global feature information or image global feature information. Target feature information may represent coarse and fine granularity feature information obtained by fusing multiple feature information, including text coarse and fine granularity fusion features or image coarse and fine granularity fusion features.

[0050] For example, the text feature information is used to perform weighted processing on the local feature information of the text to obtain the fine-grained feature information of the text; thereby, the fine-grained feature information of the text is fused and weighted with the global feature information of the text (which can be considered as the coarse-grained feature information of the text) to obtain the coarse-grained and fine-grained fusion feature of the text.

[0051] For example, based on the image feature information, the local feature information of the image is weighted to obtain the fine-grained feature information of the image; thereby, the fine-grained feature information of the image is fused and weighted with the global feature information of the image (which can be considered as the coarse-grained feature information of the image) to obtain the coarse-grained and fine-grained fusion features of the image.

[0052] In operation S203, an object whose target feature information satisfies a similarity condition with the object to be processed is determined as a result object corresponding to the object to be processed, and the modality of the object to be processed is different from the modality of the result object, and the modality includes at least one of text, image, video and voice.

[0053] In the embodiments of the present disclosure, the similarity condition can be used to determine the conditional threshold of the similarity between the object to be processed and the initial result object. Based on the comparison result of the conditional threshold, an object that meets the conditional threshold can be selected from multiple initial result objects as the result object. The specific conditional threshold can be determined according to the actual retrieval or recognition accuracy requirements and is not limited here. It is understood that the modality of the object to be processed may be different from the modality of the processing result.

[0054] For example, an image whose coarse and fine granularity fusion features satisfy a conditional threshold may be determined as a retrieval result of text retrieval image, or a text whose coarse and fine granularity fusion features satisfy a conditional threshold may be determined as a retrieval result of image retrieval text.

[0055] In another feasible embodiment, the modality of the object to be processed and the modality of the processing result may also be the same. For example, in an application scenario where a first image is used to retrieve a second image, a second image whose coarse and fine granularity fusion features with the first image meet a conditional threshold can be determined as the retrieval result of the first image.

[0056] According to the embodiments of the present disclosure, by fusing the key feature information, local feature information and global feature information of the object to be processed, the target feature information containing coarse and fine granularity fusion is obtained, thereby determining the result object corresponding to the object to be processed. Since the target feature information is determined based on the key feature information (target frame information or text feature information), on the basis of retaining the original global features, the extracted target frame information or text feature information is added with fine-grained features, which enhances the perception of fine-grained image and text information in cross-modal retrieval, improves the representation ability of the entire feature, and further enhances the user experience.

[0057] It can be understood that the above has explained how to determine the result object, and the following will explain how to determine the target feature information.

[0058] According to an embodiment of the present disclosure, target feature information is obtained based on key feature information, local feature information and global feature information, including: processing local feature information based on key feature information to obtain fine-grained feature information; and fusing the fine-grained feature information and global feature information to obtain target feature information.

[0059] In the embodiments of the present disclosure, based on the different modalities of the object to be processed, the fine-grained feature information may include text fine-grained features or image fine-grained features. The fine-grained feature information may be obtained by weighted fusion of local feature information using key feature information. Based on the different modalities of the object to be processed, the target feature information may include target text features (text coarse-grained and fine-grained fusion features) or target image features (image coarse-grained and fine-grained fusion features), and the target feature information may be obtained by further weighted fusion of global feature information with the fine-grained feature information.

[0060] In the embodiments of the present disclosure, both the fine-grained feature information and the global feature information contain weight information of their respective objects. For example, the weight information corresponding to the fine-grained feature information is the first weight, and the weight information corresponding to the global feature information is the second weight. The sum of the weights of the first weight and the second weight satisfies the target weight value, and the target weight value is a fixed value. The initial weight can characterize the initial value of the weight of the fine-grained feature. The first weight can be obtained by updating the initial weight. It should be noted that the initial weight can be determined based on actual experience or relevant test results.

[0061] Taking an image as an example, after determining the key feature information (target frame information) of the image, the target frame information can be used to perform weighted processing on the local feature information of the image (such as image blocks) to obtain fine-grained features of the image.

[0062] In the embodiments of the present disclosure, the local feature information of the image can be extracted through the information recognition layer (coding layer) of the multimodal model. For example, the image to be processed is divided into multiple image blocks, and the multiple image blocks are input into the coding layer of the multimodal model (which may include a convolution layer and a local response layer); the image block features are extracted through the convolution layer, and the convolution kernel can capture the initial local features of the image block through a sliding window operation, such as texture, edge, color, etc.; and then the initial local features are abstracted in the high-level convolution through the local response layer to obtain local feature information (such as local object parts, shape or texture information).

[0063] Taking the object to be processed as text as an example, after determining the key feature information of the text (text feature information), the text feature information can be used to perform weighted processing on the local features of the text to obtain fine-grained features of the text.

[0064] In the embodiments of the present disclosure, local feature information of the text can be extracted through the encoding layer of the multimodal model. For example, starting from word embedding, the multimodal model maps each word in the text to a high-dimensional vector space, and these word vectors capture the semantic information of the word itself (such as word meaning, context, and collocation relationship); further processing is performed through the attention mechanism, so that the representation of each word not only depends on the embedding of the word itself, but also can be weighted and adjusted according to the context, so as to capture richer local semantic information and obtain local features of the text.

[0065] In an embodiment of the present disclosure, a method for determining target feature information may include: using a first weight corresponding to the fine-grained feature information to update the initial weight of the global feature information, and obtaining a second weight after the global feature information is updated; thereby obtaining the target feature information by multiplying the first weight by the fine-grained feature information, and the second weight by the global feature information. Based on the different modalities of the object to be processed, the target feature information may include image coarse and fine granularity fusion features (image target feature information) or text coarse and fine granularity fusion features (text target feature information).

[0066] It can be understood that an example of how to determine target feature information has been described above, and an example of how to determine fine-grained feature information will be described below.

[0067] According to an embodiment of the present disclosure, when the object to be processed is an image or a video, the key feature information is the target frame information, and the target frame information is obtained by detecting the object to be processed; the local feature information is processed based on the key feature information to obtain fine-grained feature information, including: determining the fine-grained feature information based on overlapping image information between the local feature information and the target frame information.

[0068] In the embodiments of the present disclosure, the target frame information can represent the detection frame position information and the object category information in the detection frame obtained by the information detection layer. The information detection layer is used to detect the target object from the image and obtain its position in the image (i.e., the coordinates of the detection frame) and the object category. It can be understood that the information detection layer can also be an information detection submodel in the multimodal model.

[0069] For example, by inputting the object to be processed into the information detection layer, a tensor can be output, which represents the relevant information of each detection box in the image, including the bounding box coordinates, including the center coordinates (x, y) of each box, and the width w and height h of the box; the confidence value of each detection box (indicating the probability that the target object exists in the box) and the category probability can also be obtained, which can represent the probability of each category in the box.

[0070] For example, after obtaining local feature information of an image (such as local object parts, shapes or texture information), fine-grained feature information of the image can be obtained based on the local object shape, detection frame coordinates, object category and confidence value. After obtaining fine-grained feature information of an image, target feature information of the image (coarse-grained and fine-grained fusion feature information of the image) can be obtained based on the fine-grained feature information of the image, the local feature information of the image and the global feature information of the image.

[0071] Reference below Figure 3A , a method for determining fine-grained fusion feature information of an image is described in conjunction with a specific embodiment.

[0072] Figure 3A An example diagram schematically shows a process for determining fine-grained fusion feature information of an image according to an embodiment of the present disclosure.

[0073] like Figure 3A As shown, in 300A, the object to be processed is image 301. The information detection layer 302 in the multimodal model is used to detect the image 301 to obtain target frame information 303, which includes the target frame coordinates and the object category in the detection frame (for example, dog, bicycle); the information recognition layer 304 is used to recognize the image 301 to obtain image local feature information 305 and image global feature information 306 corresponding to the image 301.

[0074] After obtaining the target frame information 303, the image local feature information 305 and the image global feature information 306, the image local feature information 305 can be weighted using the target frame information 303 to obtain the image fine-grained feature 307, thereby fusing the image global feature information 306 and the image fine-grained feature 307 to obtain the image coarse-grained and fine-grained fused feature (image target feature information) 308.

[0075] According to the embodiments of the present disclosure, by utilizing the target box information to process local features, the multimodal model can refinely extract fine-grained features in the image, thereby improving the model's perception of image details. The fine-grained features improve the performance of tasks such as image classification and target detection, and achieve higher accuracy and stronger expressiveness in multimodal fusion tasks.

[0076] It can be understood that an example of how to determine fine-grained feature information has been described above, and another example of how to determine fine-grained feature information will be described below.

[0077] According to an embodiment of the present disclosure, when the object to be processed is speech or text, the key feature information is text feature information; local feature information is processed based on the key feature information to obtain fine-grained feature information, including: determining the fine-grained feature information based on overlapping text information between the local feature information and the text feature information.

[0078] In an embodiment of the present disclosure, the key feature information corresponding to the text is text feature information, and the method for obtaining the text feature information may include: performing word segmentation processing on the text to obtain word segmentation features, and retrieving the text to obtain a candidate image set (image library), thereby using the information detection layer to detect the images in the candidate image set to obtain text information corresponding to the text, and on this basis, performing intersection processing on the text information and the word segmentation features to determine the overlapping text information, and if the word segmentation is in the intersection, the weight of the word segmentation is increased to obtain text feature information. It can be understood that the text information corresponding to the text can be the classification information corresponding to the object in the image obtained by image detection. After determining the text feature information, the text feature information can be weighted fused with the local features of the text to obtain the coarse and fine granularity fused feature information of the text.

[0079] Reference below Figure 3B , the method for determining the coarse and fine granularity fusion feature information of the text is further explained in conjunction with a specific embodiment.

[0080] Figure 3B An example diagram schematically shows a process for determining text coarse-grained and fine-grained fusion feature information according to an embodiment of the present disclosure.

[0081] like Figure 3B As shown, in 300B, the object to be processed is text 310, and the text 310 is segmented to obtain the segmentation feature 311, and the text 310 is retrieved to obtain a candidate image set (image library); then the information detection layer 302 is used to detect the candidate image 312 in the candidate image set to obtain text information 313 corresponding to the text 310, and on this basis, the text information 313 is intersectionally (overlapped) with the segmentation feature 311 to obtain text feature information 314.

[0082] The text 310 is recognized using the information recognition layer 304 to obtain text local feature information 315 and text global feature information 316 corresponding to the text 310 .

[0083] After obtaining the text feature information 314, the text local feature information 315 and the text global feature information 316, the text feature information 314 can be used to perform weighted processing on the text local feature information 315 to obtain the text fine-grained feature 317, thereby fusion processing the text global feature information 316 and the text fine-grained feature 317 to obtain the text coarse-grained and fine-grained fusion feature (text target feature information) 318.

[0084] According to an embodiment of the present disclosure, through the overlapping text information between local feature information and text feature information, key semantic information can be captured, enabling the model to more accurately understand the relationships between different text parts, thereby better handling complex text tasks. For example, in multi-topic text classification, by combining local features and text feature information, the model can better understand and distinguish different topics in the text. Especially in texts with multiple topics or similar topics, the discrimination of classification can be enhanced through fine-grained features.

[0085] According to an embodiment of the present disclosure, the method further includes: performing word segmentation processing on speech or text to obtain a plurality of word segmentation features; and determining, from the plurality of word segmentation features, the word segmentation features that are the same as the text information corresponding to the candidate image as the text feature information, where the candidate image is retrieved based on text or speech.

[0086] In an embodiment of the present disclosure, according to the difference in the types in the input information, the acquisition method of the word segmentation features is different. For example, when the object to be processed is text, the acquisition method of the word segmentation features may include: preprocessing the text, for example, removing irrelevant characters (such as high-frequency but unimportant words for analysis like "de", "le", "zai", etc.); using a word segmentation tool (supporting multiple languages) to perform word segmentation processing on the text to obtain word segmentation features. It can be understood that different modes (such as exact mode, full mode, etc.) can be selected according to the actual situation to meet the requirements of different word segmentation tasks.

[0087] It can be understood that when the object to be processed is speech, the acquisition method of the word segmentation features needs to process and recognize the speech information in the early stage (such as using Mel-frequency cepstral coefficients to extract the features of the speech signal and automatic speech recognition technology) to convert the speech signal into text. The subsequent process is the same as that of text processing and will not be elaborated here.

[0088] In an embodiment of the present disclosure, the candidate image may be a plurality of candidate images obtained by recognizing and retrieving the text, and is used for the subsequent determination of the text feature information. The acquisition method and means of the candidate image are not limited herein.

[0089] For example, given a text query, the query text can be encoded into a text vector; at the same time, the system can select some images from a large image library or candidate set as a candidate image set. The candidate image set can be screened out through previous calculations or simple image search methods (such as quick screening based on image metadata, tags, etc.), and then obtain an initial candidate image set; for each initial candidate image, calculate the image-text similarity between the image vector and the query text vector, and when the image-text similarity meets the preset threshold, determine the initial image with the highest similarity to the query text as the candidate image; after determining the candidate image, text information corresponding to the candidate image can be generated (such as object classification information in the image). After determining the text information and word segmentation features, make an intersection between the text information and the word segmentation features. If the word segmentation is in the intersection, increase the weight of the word segmentation feature to obtain text feature information.

[0090] It can be understood that the above has further explained an example of how to determine text feature information, and the following will explain how to determine target feature information.

[0091] According to an embodiment of the present disclosure, fine-grained feature information includes fine-grained features and a first weight corresponding to the fine-grained features; the fine-grained feature information and the global feature information are fused to obtain target feature information, including: updating the initial weight of the global feature information based on the first weight to obtain a second weight corresponding to the global feature information, and the sum of the first weight and the second weight satisfies the target weight value; and weighted fusion of the fine-grained features and the global feature information based on the first weight and the second weight, respectively, to determine the target feature information.

[0092] In an embodiment of the present disclosure, the initial weight of the global feature information is updated to obtain a second weight corresponding to the global feature information, the weight corresponding to the fine-grained feature is the first weight, and the target weight value can be the sum of the first weight and the second weight (a fixed value); thereby, the product result of the first weight and the fine-grained feature, as well as the product result of the second weight and the global feature information can be calculated, and the target feature information can be determined on this basis.

[0093] For example, the initial weight corresponding to the global feature information is α, and the weight corresponding to the fine-grained feature is determined to be the first weight, denoted by β. The specific values ​​of α and β can be determined based on experiments or experience; the initial weight α is updated based on the first weight β and the target weight value to obtain the updated second weight value α', and the updated α' satisfies α'+β=1; on this basis, the fine-grained feature (denoted by F local ) is weighted, and the second weight α' is used to weight the global feature information (denoted as F global ) is weighted to obtain the target feature information, that is, the target feature information = β*F local+α'*F global .

[0094] According to the embodiments of the present disclosure, by fusing fine-grained features and global feature information, the local details and global structure of the object to be processed can be better captured. The fine-grained features focus on the local details of the object to be processed (such as local shape, texture, etc.), while the global features focus on the overall structure of the object to be processed. The combination of the two helps to accurately identify the result object in complex backgrounds or multiple targets.

[0095] It can be understood that the above has further explained an example of how to determine the target feature information, and the training process of the deep learning model used by the object processing method will be explained below.

[0096] According to an embodiment of the present disclosure, the object processing method is executed by a deep learning model, and the deep learning model includes an information detection layer and an information recognition layer; the training process of the deep learning model includes: inputting the first sample object to be processed into the information detection layer in the deep learning model to obtain sample target frame information; inputting the first sample object to be processed and the second sample object to be processed into the information recognition layer to obtain first sample local feature information, first sample global feature information and second sample local feature information, second sample global feature information; the modality of the first sample object to be processed is different from the modality of the second sample object to be processed.

[0097] In an embodiment of the present disclosure, the deep learning model may be a cross-modal retrieval model, such as a Contrastive Language-Image Pretraining (CLIP) model. The information detection layer may represent a first network layer for detecting image frames in sample images or sample videos to obtain key feature information; the information recognition layer may represent a second network layer for extracting features from the object to be processed and identifying the initial feature information.

[0098] In an embodiment of the present disclosure, the first sample object to be processed (such as a sample image or a sample video frame) is detected and processed using an information detection layer to obtain sample target frame information including detection frame (target frame) coordinates, confidence and category probability; the first sample object to be processed is feature extracted using an information recognition layer to obtain first sample local feature information and first sample global feature information; and the second sample object to be processed (such as a sample text or sample audio) is feature extracted using an information recognition layer to obtain second sample local feature information and second sample global feature information.

[0099] According to an embodiment of the present disclosure, a deep learning model is trained by a loss function determined by the following operations: determining first sample key feature information and second sample key feature information corresponding to a first sample object to be processed and a second sample object to be processed respectively based on sample target frame information; determining sample target image feature information and sample target text feature information based on the first sample key feature information, the second sample key feature information, the first sample local feature information, the first sample global feature information, the second sample local feature information and the second sample global feature information to obtain sample feature pairs; determining a loss function based on the similarity between sample feature pairs and label similarity; label similarity is obtained based at least on the original similarity between the first sample object to be processed and the second sample object to be processed.

[0100] It can be understood that the sample local feature information includes the first sample local feature information and the second sample local feature information, and the sample global feature information includes the first sample global feature information and the second sample global feature information. It should be noted that the first sample and the second sample are only used to distinguish the objects to be processed in different modalities, and the specific representation objects are not limited here.

[0101] In the embodiments of the present disclosure, the loss function can be used to measure the matching degree between samples of different modalities in the multimodal model. The label similarity can be determined by the original similarity between the actual different sample objects to be processed. For example, the label similarity can be (0, 1), where 0 represents the lowest matching degree or similarity between samples of different modalities, and 1 represents the highest matching degree or similarity between samples of different modalities.

[0102] Taking the first sample to be processed object as a sample image and the second sample to be processed object as a sample text as an example, the first sample key feature information may be the sample target frame information corresponding to the sample image, and the second sample key feature information may be the sample text feature information corresponding to the sample text; thereby, the sample target image feature information is determined based on the sample target frame information, the sample image local feature information and the sample image global feature information, and the sample target text feature information is determined based on the sample text feature information, the sample text local feature information and the sample text global feature information, and on this basis, the sample feature pairs are constructed using the sample target image feature information and the sample target text feature information; then, the similarity between the sample feature pairs is calculated using a similarity calculation strategy (such as cosine similarity), and the similarity value is converted into a probability value, which may reflect the possibility of a match between the sample image and the sample text, for example, the Sigmoid function may be used to convert the similarity into a probability, denoted as p; then, the cross entropy strategy may be used to calculate the difference between the predicted probability (p) and the label similarity (0 or 1), and the loss function is obtained by minimizing the error.

[0103] In an embodiment of the present disclosure, the sample target frame information can represent the sample detection frame position information and the sample object category information in the sample detection frame corresponding to the first sample object to be processed and the second sample object to be processed obtained by the information detection layer of the deep learning model.

[0104] Figure 3C An example diagram of the training process of a deep learning model according to an embodiment of the present disclosure is schematically shown.

[0105] like Figure 3C As shown, in 300C, the first sample object to be processed is a sample image 411, and the second sample object to be processed is a sample text 412. The sample image 411 is detected by using the information detection layer 41 in the initial multimodal model to obtain sample target frame information 413 and text information 418 corresponding to the object in the target frame. The sample target frame information 413 includes sample target frame coordinates and object category information in the sample detection frame; the sample image 411 is identified by using the information recognition layer 42 in the initial multimodal model to obtain sample image local feature information 414 and sample image global feature information 415 corresponding to the sample image 411.

[0106] After obtaining the sample target frame information 413, the sample image local feature information 414 and the sample image global feature information 415, the sample target frame information 413 can be used to perform weighted processing on the sample image local feature information 414 to obtain the sample image fine-grained feature 416, thereby fusion processing the sample image global feature information 415 and the sample image fine-grained feature 416 to obtain the sample target image feature information (sample image coarse-grained and fine-grained fusion feature) 417.

[0107] The sample text 412 is segmented to obtain a sample segmentation feature 421, and the sample segmentation feature 421 is intersected with the text information 418 to obtain overlapping text information. If the sample segmentation feature is in the overlapping text information, the weight of the sample segmentation feature is increased as the sample text feature information 422. The sample text 412 is identified by the information identification layer 42 to obtain sample text local feature information 423 and sample text global feature information 424 corresponding to the sample text 412.

[0108] After obtaining the sample text feature information 422, the sample text local feature information 423 and the sample text global feature information 424, the sample text feature information 422 can be used to perform weighted processing on the sample text local feature information 423 to obtain the sample text fine-grained feature 425, thereby fusion processing the sample text global feature information 424 and the sample text fine-grained feature 425 to obtain the sample text target feature information (sample text coarse-grained and fine-grained fusion feature information) 426.

[0109] After obtaining the sample target image feature information 417 and the sample text target feature information 426, a sample feature pair 430 can be obtained, thereby determining a loss function 440 based on the similarity between the sample feature pairs 430 and the label similarity, and further obtaining a trained target multimodal model 450.

[0110] Based on the above object processing method, the present disclosure also provides an object processing device. Figure 4 The device is described in detail.

[0111] Figure 4 A block diagram of an object processing device according to an embodiment of the present disclosure is shown.

[0112] like Figure 4 As shown, the object processing device 400 includes an information output module 401 , a target feature information obtaining module 402 and a result generating module 403 .

[0113] According to some embodiments of the present disclosure, the object processing device 400 may be used to implement the object processing method of the embodiments of the present disclosure.

[0114] The information output module 401 may perform, for example, operation S201 to process the object to be processed and obtain key feature information and initial feature information of the object to be processed, where the initial feature information includes local feature information and global feature information.

[0115] The target feature information obtaining module 402 can execute, for example, operation S202, which is used to fuse the key feature information, local feature information and global feature information to obtain the target feature information of the object to be processed. The key feature information is target frame information or text feature information of the object to be processed based on different modalities.

[0116] The result generation module 403 can execute, for example, operation S203, which is used to determine that an object whose target feature information satisfies a similarity condition with the object to be processed is a result object corresponding to the object to be processed, and the modality of the object to be processed is different from the modality of the result object, and the modality includes at least one of text, image, video and voice.

[0117] According to the embodiment of the present disclosure, based on the information output module 401, the target feature information acquisition module 402 and the result generation module 403 in the object processing device 400, the key feature information, local feature information and global feature information of the object to be processed are fused to obtain the target feature information containing coarse and fine granularity fusion, thereby determining the result object corresponding to the object to be processed. Since the target feature information is determined based on the key feature information (target frame information or text feature information), on the basis of retaining the original global features, the extracted target frame information or text feature information is increased with fine-grained features, which enhances the perception of fine-grained image and text information in cross-modal retrieval, improves the representation ability of the entire feature, and further enhances the user experience.

[0118] It is understandable that the information output module 401, the target feature information obtaining module 402 and the result generation module 403 can be combined in one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present invention, at least one of the information output module 401, the target feature information obtaining module 402 and the result generation module 403 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented in hardware or firmware such as any other reasonable way of integrating or encapsulating the circuit, or in a proper combination of software, hardware and firmware. Alternatively, at least one of the information output module 401, the target feature information obtaining module 402 and the result generation module 403 can be at least partially implemented as a computer program module, and when the program is run by a computer, the function of the corresponding module can be executed.

[0119] According to an embodiment of the present disclosure, the target feature information obtaining module 402 includes: a local feature information processing submodule and a fusion processing submodule. The local feature information processing submodule is used to process the local feature information based on the key feature information to obtain fine-grained feature information; and the fusion processing submodule is used to fuse the fine-grained feature information and the global feature information to obtain the target feature information.

[0120] According to an embodiment of the present disclosure, when the object to be processed is an image or a video, the key feature information is the target frame information, and the target frame information is obtained by detecting the object to be processed; the local feature information processing submodule includes: a fine-grained feature information determination unit, which is used to determine the fine-grained feature information based on the overlapping image information between the local feature information and the target frame information.

[0121] According to an embodiment of the present disclosure, when the object to be processed is speech or text, the key feature information is text feature information; the local feature information processing submodule includes: a fine-grained feature information determination unit, which is used to determine the fine-grained feature information based on the overlapping text information between the local feature information and the text feature information.

[0122] According to an embodiment of the present disclosure, the device further includes: a word segmentation processing module and a text feature information determination module. The word segmentation processing module is used to perform word segmentation processing on speech or text to obtain multiple word segmentation features; and the text feature information determination module is used to determine the word segmentation feature that is the same as the text information corresponding to the candidate image from the multiple word segmentation features as the text feature information, and the candidate image is retrieved based on the text or speech.

[0123] According to an embodiment of the present disclosure, the fine-grained feature information includes the fine-grained feature and a first weight corresponding to the fine-grained feature; the fusion processing submodule includes: a second weight obtaining unit and a target feature information determining unit. The second weight obtaining unit is used to update the initial weight of the global feature information based on the first weight, obtain the second weight corresponding to the global feature information, and the sum of the first weight and the second weight satisfies the target weight value; and the target feature information determining unit is used to perform weighted fusion on the fine-grained feature and the global feature information based on the first weight and the second weight, respectively, to determine the target feature information.

[0124] According to an embodiment of the present disclosure, the object processing method is executed by a deep learning model, and the deep learning model includes an information detection layer and an information recognition layer; the training process of the deep learning model includes: inputting the first sample object to be processed into the information detection layer in the deep learning model to obtain sample target frame information; and inputting the first sample object to be processed and the second sample object to be processed into the information recognition layer to obtain first sample local feature information, first sample global feature information and second sample local feature information, second sample global feature information; the modality of the first sample object to be processed is different from the modality of the second sample object to be processed.

[0125] According to an embodiment of the present disclosure, a deep learning model is trained by a loss function determined by the following operations: determining first sample key feature information and second sample key feature information corresponding to a first sample object to be processed and a second sample object to be processed respectively based on sample target frame information; determining sample target image feature information and sample target text feature information based on the first sample key feature information, the second sample key feature information, the first sample local feature information, the first sample global feature information, the second sample local feature information and the second sample global feature information to obtain sample feature pairs; determining a loss function based on the similarity between sample feature pairs and label similarity; label similarity is obtained based at least on the original similarity between the first sample object to be processed and the second sample object to be processed.

[0126] Figure 5 A schematic block diagram of an example electronic device that can be used to implement the object processing method according to an embodiment of the present disclosure is shown.

[0127] Electronic devices are intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0128] like Figure 5 As shown, the device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0129] A number of components in the electronic device 500 are connected to the I / O interface 505, including: an input unit 506, such as a keyboard, a mouse, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a disk, an optical disk, etc.; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 allows the device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0130] The computing unit 501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as the virtual image driving method. For example, in some embodiments, the virtual image driving method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, one or more steps of the virtual image driving method described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform the virtual image driving method in any other appropriate manner (e.g., by means of firmware).

[0131] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0132] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0133] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0134] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0135] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0136] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. Among them, the server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0137] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations and / or combinations are not explicitly described in the present disclosure. In particular, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways without departing from the spirit and teachings of the present disclosure. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0138] Although the present disclosure has been shown and described with reference to specific exemplary embodiments of the present disclosure, it should be understood by those skilled in the art that various changes in form and details may be made to the present disclosure without departing from the spirit and scope of the present disclosure as defined by the appended claims and their equivalents. Therefore, the scope of the present disclosure should not be limited to the above-mentioned embodiments, but should be determined not only by the appended claims, but also by the equivalents of the appended claims.

Claims

1. An object processing method, comprising: Processing the object to be processed to obtain key feature information and initial feature information of the object to be processed, wherein the initial feature information includes local feature information and global feature information; The key feature information, the local feature information and the global feature information are fused to obtain target feature information of the object to be processed, wherein the key feature information is target frame information or text feature information of the object to be processed based on different modalities; as well as An object whose target feature information satisfies a similarity condition with that of the object to be processed is determined as a result object corresponding to the object to be processed, wherein the modality of the object to be processed is different from the modality of the result object, and the modality includes at least one of text, image, video and voice.

2. The method according to claim 1, wherein: Obtaining target feature information based on the key feature information, the local feature information and the global feature information includes: Processing the local feature information based on the key feature information to obtain fine-grained feature information; and The fine-grained feature information and the global feature information are fused to obtain the target feature information.

3. The method according to claim 2, wherein: In the case where the object to be processed is an image or a video, the key feature information is target frame information, and the target frame information is obtained by detecting the object to be processed; The local feature information is processed based on the key feature information to obtain fine-grained feature information, including: The fine-grained feature information is determined based on overlapping image information between the local feature information and the target frame information.

4. The method according to claim 2, wherein: When the object to be processed is speech or text, the key feature information is the text feature information; The local feature information is processed based on the key feature information to obtain fine-grained feature information, including: The fine-grained feature information is determined based on overlapping text information between the local feature information and the text feature information.

5. The method according to claim 4, further comprising: Performing word segmentation processing on the speech or the text to obtain multiple word segmentation features; as well as A word segmentation feature identical to text information corresponding to a candidate image is determined from the plurality of word segmentation features as the text feature information, and the candidate image is retrieved based on the text or the voice.

6. The method according to claim 2, wherein: The fine-grained feature information includes a fine-grained feature and a first weight corresponding to the fine-grained feature; The fine-grained feature information and the global feature information are fused to obtain the target feature information, including: Based on the first weight, an initial weight of the global feature information is updated to obtain a second weight corresponding to the global feature information, wherein a sum of the first weight and the second weight satisfies a target weight value; as well as The fine-grained features and the global feature information are weightedly fused based on the first weight and the second weight to determine the target feature information.

7. The method according to claim 1, wherein: The object processing method is performed by a deep learning model, and the deep learning model includes an information detection layer and an information recognition layer; The training process of the deep learning model includes: Inputting the first sample to-be-processed object into the information detection layer in the deep learning model to obtain sample target frame information; The first sample object to be processed and the second sample object to be processed are input into the information recognition layer to obtain the first sample local feature information, the first sample global feature information and the second sample local feature information and the second sample global feature information; the modality of the first sample object to be processed is different from the modality of the second sample object to be processed.

8. The method according to claim 7, wherein: The deep learning model is trained by a loss function determined by the following operations: Determine, based on the sample target frame information, first sample key feature information and second sample key feature information corresponding to the first sample to-be-processed object and the second sample to-be-processed object, respectively; Determine sample target image feature information and sample target text feature information based on the first sample key feature information, the second sample key feature information, the first sample local feature information, the first sample global feature information, the second sample local feature information, and the second sample global feature information to obtain a sample feature pair; Determining the loss function based on the similarity between the sample feature pairs and the label similarity; The label similarity is obtained based on at least the original similarity between the first sample object to be processed and the second sample object to be processed.

9. An object processing device, comprising: An information output module is used to process the object to be processed to obtain key feature information and initial feature information of the object to be processed, wherein the initial feature information includes local feature information and global feature information; A target feature information obtaining module is used to fuse the key feature information, the local feature information and the global feature information to obtain the target feature information of the object to be processed, wherein the key feature information is target frame information or text feature information of the object to be processed based on different modalities; as well as A result generation module is used to determine that an object whose target feature information satisfies a similarity condition with the object to be processed is a result object corresponding to the object to be processed, and the modality of the object to be processed is different from the modality of the result object, and the modality includes at least one of text, image, video and voice.

10. An electronic device comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 8.

Images