Multimodal feature fusion method, device, equipment, medium and product

Abstract

The present disclosure provides a method, device, equipment, medium and product for fusing multimodal features, which relate to the field of image processing technology, and in particular to the field of computer vision and deep learning technology. The specific implementation scheme is: obtaining multiple modal features of the same image; encoding each of the multiple modal features; adopting a multi-head attention mechanism to perform feature cross-fusion on all encoded features obtained by encoding to obtain multiple initial fusion features; and fusing the multiple initial fusion features again to obtain target fusion features. The present disclosure adopts a multi-head attention mechanism to cross-fuse the encoding features of multiple modal features, so that the encoding features can be fully fused. The multiple initial fusion features generated by the cross-fusion are fused again, and the target fusion features finally obtained can accurately describe the POI features in the image, thereby improving the recall rate.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to the field of computer vision and deep learning technology. Background Art

[0002] In the process of producing points of interest (POI), the multimodal features of the same sign image are fused to improve the recall rate. However, if the multimodal features are fused by splicing them together in the related technology, the fused features obtained not only have redundant information, but also have low discriminability, which leads to a lower recall rate and cannot be used in production. Summary of the invention

[0003] The present disclosure provides a method, device, equipment, medium and product for fusing multimodal features.

[0004] According to one aspect of the present disclosure, a method for fusing multimodal features is provided, comprising: acquiring multiple modal features of the same image; encoding each of the multiple modal features; adopting a multi-head attention mechanism to perform feature cross-fusion on all encoded features to obtain multiple initial fused features; and fusing the multiple initial fused features again to obtain a target fused feature.

[0005] According to another aspect of the present disclosure, a multimodal feature fusion device is provided, including: an acquisition unit, used to acquire multiple modal features of the same image; an encoding unit, used to encode each of the multiple modal features; a first fusion unit, used to adopt a multi-head attention mechanism to perform feature cross-fusion on all encoded features obtained by encoding to obtain multiple initial fusion features; and a second fusion unit, used to fuse the multiple initial fusion features again to obtain a target fusion feature.

[0006] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the described method.

[0007] According to yet another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method.

[0008] According to yet another aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program implements the method when executed by a processor.

[0009] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0011] Figure 1 It is a schematic diagram of the main process of automated production provided according to an embodiment of the present disclosure;

[0012] Figure 2 is a flow chart of a method for fusing multimodal features provided according to an embodiment of the present disclosure;

[0013] Figure 3 is a block diagram of performing multimodal feature fusion according to an embodiment of the present disclosure;

[0014] Figure 4 is a block diagram of a multi-modal feature fusion device according to an exemplary embodiment;

[0015] Figure 5 It is a block diagram of an electronic device used to implement the multimodal feature fusion method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0016] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0017] The present disclosure is applied to the scenario of producing map POI data, specifically, the scenario of attaching a signboard image to a POI library. Figure 1 Schematic diagram of the main process of automated production provided according to the embodiment of the present disclosure. Figure 1As shown, the main process of automated production of POI includes: performing optical character recognition (OCR) on the signboard image to obtain the text features of the signboard image. Input the signboard image into the image feature extraction network for image feature extraction. The image feature extraction network can be a VIT (Vision Transformer) network or a deep residual network (ResNet). Distance indexing is performed based on the coordinates of the POI in the signboard image. It can be understood that the POI name and POI image are recalled within 50 meters of the coordinates of the POI in the signboard image. The recalled POI name and POI image are respectively matched with the extracted text features and image features for text and image matching. The process of text matching and image matching can be understood as feature similarity retrieval. Integrate the above matching results to obtain the hanging result of the signboard image. The signboard image contains POI, so the signboard image is also called POI image. Figure 1 The img in the representation represents the signboard image.

[0018] The defects of the signboard image mounting result obtained according to the above process include that the use of image features, text features and spatial coordinate information to recall data separately increases the complexity and calculation amount of the entire system, resulting in low production efficiency. On the other hand, single-modal information cannot accurately characterize POI features, resulting in recall loss. The above defects can be solved by fusing multi-modal features such as image features, text features and POI coordinates. However, feature splicing cannot effectively fuse features of multiple modes, and the obtained fused features not only have redundant information but also lack discriminability. The fused features obtained by splicing have a low recall rate and cannot be directly applied in production.

[0019] In view of this, the present disclosure provides a multimodal feature fusion method. The multimodal features are cross-fused through a multi-head attention mechanism to obtain effective and more discriminative fusion features. The fusion features obtained by the present disclosure are used in the automated production of POIs, which can improve production efficiency and recall rate.

[0020] The following embodiments of the present disclosure will illustrate the method for fusing multimodal features in conjunction with the accompanying drawings.

[0021] Figure 2 FIG. 1 is a flow chart of a method for fusing multimodal features according to an embodiment of the present disclosure. Figure 2 As shown, the multimodal feature fusion method provided by the present disclosure includes the following steps.

[0022] In step S201, multiple modality features of the same image are obtained.

[0023] Modal features include image features, text features, and POI coordinates in the image. In addition to image features, text features, and POI coordinates in the image, modal features can also include attribute information of the POI in the image, such as the shooting angle of the image, the time when the image was taken, and the area where the image is located.

[0024] The present disclosure can not only fuse image features and text features, but also fuse the shooting angle, shooting time and image area of ​​the image, so that the fused features can describe the image more accurately.

[0025] Image features can be obtained through an existing image feature extraction network. The image feature extraction network for obtaining image features can be a deep convolutional neural network (CNN). Text features can be obtained by performing OCR recognition on the image. In addition to obtaining text features through OCR recognition, BERT (Bidirectional Encoder Representations from Transformers), ERNIE and other networks can also be used to obtain text features. It should be noted that the present disclosure does not limit the method of obtaining each modal feature.

[0026] In step S202, each modal feature among the multiple modal features is encoded.

[0027] Each modal feature is encoded so that it has the same dimension without losing the original information.

[0028] In step S203, a multi-head attention mechanism is used to perform feature cross-fusion on all encoded features to obtain multiple initial fused features.

[0029] The encoded features are assigned to the parameters in the multi-head attention mechanism, and the multi-head attention mechanism is used to cross-fuse the encoded features to obtain multiple initial fused features. Compared with related technologies, the multi-head attention mechanism can fully fuse the encoded features instead of simply splicing them.

[0030] In step S204, the multiple initial fusion features are fused again to obtain the target fusion feature.

[0031] The disclosed embodiment inputs multiple initial features into a multilayer perceptron (MLP) for re-fusion to obtain a target fusion feature. The target fusion feature obtained by re-fusion of multiple initial fusion features using a multilayer perceptron can accurately describe POI features.

[0032] In the embodiments of the present disclosure, other network models besides MLP can be used to fuse multiple initial features, and are not limited to using MLP.

[0033] The present invention encodes multiple modal features, and cross-fuses the encoded features using a multi-head attention mechanism, so that the encoded features can be fully fused. The multiple initial fusion features generated by the cross-fusion are fused again, and the final target fusion features can accurately describe the POI features in the image, thereby improving the recall rate.

[0034] In the disclosed embodiments, different encoding methods are used for different modal features.

[0035] For the image features, determine the feature dimensions of the feature map that characterizes the image features. If its feature dimensions are (B, H*W, C) and meet the requirements for fusion, the image features are not further processed. If its feature dimensions do not meet the requirements for fusion, a preset neural network is used to extract image features that meet the fusion requirements from the image features. Where B represents the batch size. C represents the number of channels included in the feature map. H represents the height of the feature map. W represents the width of the feature map.

[0036] For POI coordinates, GeoHash is used to vector encode the POI coordinates, and the encoded vector is vector-upgraded. Based on the encoded vector after vector dimension upgrade, the POI coordinates are encoded. The specific process is: use GeoHash to encode the POI coordinates as 0, 1 vectors. Use vector dimension upgrade to convert the 0, 1 vector into a two-dimensional matrix vector with a dimension of (B, H*W, C). It should be noted that the vector dimension upgrade disclosed in the present invention is to perform position encoding on each 1. For example, 1110 can be encoded as: [[1, 0, 0, 0], [0, 1, 0, 0], [0, 0, 1, 0]].

[0037] The present disclosure adopts GeoHash to perform vector encoding on POI coordinates, and performs vector dimension upgrading on the encoded vectors, so that the dimensions of POI coordinates are consistent with the dimensions of other modal features, which is convenient for cross-fusion with other modal features.

[0038] For the text features, word2Vec is used for encoding, so that the dimension of the encoded text features is also (B, H*W, C). In the present disclosure, word to vector is a group of related models used to generate word vectors.

[0039] In the disclosed embodiment, other modal features except image features and POI coordinates, such as the shooting angle of the image and the time when the image was shot, are encoded using the word2vec encoding method.

[0040] The present invention uses word2vec encoding to encode all modal features except image features and POI coordinates, so that the modal features to be fused have the same dimension, in preparation for cross-fusion.

[0041] In the disclosed embodiment, for different numbers of modal features, different methods of multi-head-attention fusion are used.

[0042] In the disclosed embodiment, if the number of modal features is less than or equal to a preset number, all encoded features are directly encoded using a multi-head attention mechanism for feature cross-fusion.

[0043] In the embodiment of the present disclosure, the preset number may be 3. Since the present disclosure is aimed at fusing multi-modal features, at least two modal features are required.

[0044] In one embodiment, a multi-head attention mechanism is used to cross-fuse the coding features corresponding to the two modal features. Assume that the two modal features are image features and text features. The coding features corresponding to the image features are image coding features. The coding features corresponding to the text features are text coding features. The image coding features and the text coding features are assigned to the parameters O, K and V in the multi-head attention mechanism respectively. The first assignment method is to assign the image coding features to O, the text coding features to K, and the text coding features to V. The second assignment method is to assign the text coding features to O, the image coding features to K, and the image coding features to V. Under the above two assignment methods, multiple initial fusion features are obtained by combining them in pairs.

[0045] In another embodiment, a multi-head attention mechanism is used to cross-fuse the coding features corresponding to the three modal features. Assume that the three modal features are image features, text features, and POI coordinates. The coding features corresponding to the image features are image coding features. The coding features corresponding to the text features are text coding features. The coding features corresponding to the POI coordinates are POI coding coordinates. The image coding features, text coding features, and POI coding coordinates are assigned to the parameters O, K, and V in the multi-head attention mechanism, respectively. In the multi-head attention mechanism, each coding feature is arranged and combined in pairs to obtain the initial fusion features of 6 one-dimensional vectors. The 6 initial fusion features are fused again using MLP to obtain the target fusion features.

[0046] Through the embodiments of the present disclosure, for modal features that are less than or equal to a preset number, the multi-head attention mechanism is directly used to cross-fuse the encoding features corresponding to the modal features, thereby improving the fusion effect.

[0047] In the embodiment of the present disclosure, if the number of modal features is greater than a preset number, all encoded features obtained by encoding the modal features are combined according to the preset number; and each combination result of all the combination results obtained after the combination is subjected to feature cross-fusion using a multi-head attention mechanism.

[0048] In one embodiment, if there are 5 modal features and the preset number is 3, 3 coding features are taken out from the coding features corresponding to the 5 modal features each time and combined to obtain all the combination results. That is, there are 10 combination results in total. For each combination result, a multi-head attention mechanism is used to perform feature cross-fusion. One combination result can obtain 6 initial fusion features, and 10 combination results can obtain 60 initial fusion features. The 60 initial fusion features are input into the MLP for fusion again to obtain the target fusion feature.

[0049] In the disclosed embodiment, for modal features greater than a preset number, they are first combined according to the preset number, and then the encoding features corresponding to the modal features in each combination result are cross-fused using a multi-head attention mechanism to ensure sufficient cross-fusion of multi-modal features, making the fused features more discriminative.

[0050] In the disclosed embodiment, the network structure of the above-mentioned multimodal feature fusion method is combined, and the combined structure is trained using metric learning and other methods to generate a multimodal feature fusion model. The features obtained using the multimodal feature fusion model can meet the business application requirements.

[0051] Figure 3 is a block diagram of multimodal feature fusion according to an embodiment of the present disclosure. Figure 3 As shown, when the present disclosure is used for multimodal feature fusion, each modal feature needs to be encoded through the encoding layer in the multimodal feature fusion model. The encoded features obtained by encoding are cross-fused to obtain multiple initial fusion features. The multiple initial fusion features are input into the MLP model for fusion again to obtain the target fusion feature. Figure 3 As shown, the POI image, POI coordinates, and POI text features (POI OCR) are input into the multimodal feature fusion model. In the encoding layer of the multimodal feature fusion model, feature maps are extracted from the POI image. GeoHash encoding and vector dimension upscaling are performed on the POI coordinates. Word2Vec is used to encode the text features of the POI. The multimodal feature fusion model provided by the present disclosure has lower complexity, is easy to apply in production, and can improve efficiency while improving recall.

[0052] Based on the same concept, the embodiment of the present disclosure also provides a multi-modal feature fusion device.

[0053] It is understandable that the multimodal feature fusion device provided in the embodiment of the present disclosure includes hardware structures and / or software modules corresponding to the execution of each function in order to realize the above functions. In combination with the units and algorithm steps of each example disclosed in the embodiment of the present disclosure, the embodiment of the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the technical solution of the embodiment of the present disclosure.

[0054] Figure 4 FIG. 1 is a block diagram of a multi-modal feature fusion device according to an exemplary embodiment. Figure 4 The device 400 includes an acquisition unit 401, an encoding unit 402, a first fusion unit 403 and a second fusion unit 404.

[0055] The acquisition unit 401 is used to acquire multiple modal features of the same image; the encoding unit 402 is used to encode each of the multiple modal features; the first fusion unit 403 is used to adopt a multi-head attention mechanism to perform feature cross-fusion on all encoded features to obtain multiple initial fusion features; the second fusion unit 404 is used to fuse the multiple initial fusion features again to obtain the target fusion feature.

[0056] In one embodiment, the first fusion unit 403 is used to: if the number of modal features is less than or equal to a preset number, directly use a multi-head attention mechanism to perform feature cross-fusion on all encoded features.

[0057] In one embodiment, the first fusion unit 403 is also used for: if the number of modal features is greater than a preset number, all encoded features obtained by encoding are combined according to the preset number; and each combination result of all combination results obtained after the combination is subjected to feature cross-fusion using a multi-head attention mechanism.

[0058] In one embodiment, the modal features include: image features, text features, coordinates of points of interest (POIs) in the image, the shooting angle of the image, the time when the image was shot and / or the area where the image is located.

[0059] In one embodiment, the encoding unit 402 is used to use GeoHash to vector encode the POI coordinates if the modal feature is the POI coordinates, and to perform vector dimension upgrade on the encoded vector; and to encode the POI coordinates based on the encoded vector after vector dimension upgrade.

[0060] In one implementation, the encoding unit 402 is configured to encode the other modal features using word2Vec if the modal features are other modal features except for the image features and the POI coordinates.

[0061] In one implementation, the second fusion unit 404 is used to use a multi-layer perceptron to fuse multiple initial fusion features again to obtain a target fusion feature.

[0062] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0063] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0064] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0065] Figure 5 A schematic block diagram of an example electronic device 500 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0066] like Figure 5 As shown, the device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0067] A number of components in the device 500 are connected to the I / O interface 505, including: an input unit 506, such as a keyboard, a mouse, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a disk, an optical disk, etc.; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 allows the device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0068] The computing unit 501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 501 performs the various methods and processes described above, such as the fusion method of multimodal features. For example, in some embodiments, the fusion method of multimodal features may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, one or more steps in the fusion method of multimodal features described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to execute the multi-modal feature fusion method in any other appropriate manner (for example, by means of firmware).

[0069] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0070] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0071] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0072] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0073] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0074] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0075] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0076] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A multimodal feature fusion method, comprising: Obtain multiple modality features of the same image; Encoding each of the multiple modal features; Using a multi-head attention mechanism, all encoded features are cross-fused to obtain multiple initial fusion features; The multiple initial fusion features are fused again to obtain a target fusion feature; The multi-head attention mechanism is used to perform feature cross-fusion on all encoded features obtained by encoding, including: If the number of modal features is greater than a preset number, all encoded features obtained by encoding the modal features are combined according to the preset number; For all the combined results obtained after the combination, the multi-head attention mechanism is used to perform feature cross-fusion.

2. The method according to claim 1, wherein: The multi-head attention mechanism is used to perform feature cross-fusion on all encoded features obtained by encoding, including: If the number of modal features is less than or equal to the preset number, all encoded features are directly fused using the multi-head attention mechanism.

3. The method according to any one of claims 1 to 2, wherein: The modal features include: Image features, text features, coordinates of points of interest (POIs) in the image, the shooting angle at which the image was taken, the time at which the image was taken and / or the area where the image is located.

4. The method according to claim 3, wherein: The encoding of each modality feature among the multiple modality features comprises: If the modal feature is a POI coordinate, GeoHash is used to vector encode the POI coordinate, and the encoded vector is vector-upgraded; The POI coordinates are encoded based on the encoding vector after the vector dimension is upgraded.

5. The method according to claim 3, wherein: The encoding of each modality feature among the multiple modality features comprises: If the modal features are other modal features except image features and POI coordinates, word2Vec is used to encode the other modal features.

6. The method according to any one of claims 1 to 2, wherein: The step of fusing the multiple initial fusion features again to obtain a target fusion feature includes: The multiple initial fusion features are fused again using a multi-layer perceptron to obtain a target fusion feature.

7. A multi-modal feature fusion device, comprising: An acquisition unit, used for acquiring multiple modality features of the same image; An encoding unit, configured to encode each of the plurality of modal features; The first fusion unit is used to adopt a multi-head attention mechanism to perform feature cross-fusion on all encoded features obtained by encoding to obtain multiple initial fusion features; A second fusion unit is used to fuse the multiple initial fusion features again to obtain a target fusion feature; Wherein, the first fusion unit is further used for: If the number of modal features is greater than a preset number, all encoded features obtained by encoding the modal features are combined according to the preset number; For all the combined results obtained after the combination, the multi-head attention mechanism is used to perform feature cross-fusion.

8. The device according to claim 7, wherein: The first fusion unit is used for: If the number of modal features is less than or equal to the preset number, all encoded features are directly fused using the multi-head attention mechanism.

9. The device according to any one of claims 7 to 8, wherein: The modal features include: Image features, text features, coordinates of points of interest (POIs) in the image, the shooting angle at which the image was taken, the time at which the image was taken and / or the area where the image is located.

10. The device according to claim 9, wherein: The encoding unit is used for: If the modal feature is a POI coordinate, GeoHash is used to vector encode the POI coordinate, and the encoded vector is vector-upgraded; The POI coordinates are encoded based on the encoding vector after the vector dimension is upgraded.

11. The device according to claim 9, wherein: The encoding unit is used for: If the modal features are other modal features except image features and POI coordinates, word2Vec is used to encode the other modal features.

12. The device according to any one of claims 7 to 8, wherein: The second fusion unit is used for: The multiple initial fusion features are fused again using a multi-layer perceptron to obtain a target fusion feature.

13. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-6.

15. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images