Electromagnetic signal image-text data search method and device, equipment, medium and product

Abstract

The invention is suitable for the technical field of data search, and provides a data search method and device for an electromagnetic signal image-text, equipment, a medium and a product, and the method comprises the steps: obtaining a target electromagnetic signal image and target text content corresponding to to-be-searched data through employing an image-text generation model; extracting image semantic features of the target electromagnetic signal image and text semantic features of the target text content; fusing the image semantic feature and the text semantic feature to obtain a target image-text semantic feature; searching target reference image-text semantic features corresponding to the target image-text semantic features from a retrieval map corresponding to a knowledge base; and in the knowledge base, determining the reference electromagnetic signal image or the reference text content corresponding to the target reference image-text semantic feature as a search result of the to-be-searched data. The accuracy of searching the electromagnetic signal image is improved through searching the image-text semantic features and the retrieval atlas.

Description

Technical Field

[0001] The present application belongs to the field of data search technology, and in particular, relates to methods, devices, equipment, media and products for data search of electromagnetic signal graphics. Background Art

[0002] Electromagnetic signal image search technology is a method that uses computer vision and related algorithms to find electromagnetic signal images and text descriptions that are similar or related to a given query electromagnetic signal image from a large amount of electromagnetic signal image data. It is widely used in data search scenarios, such as graph-based knowledge management services.

[0003] Currently, electromagnetic signal image searches are mostly based on visual features, which are performed by analyzing the color, texture, shape and other visual elements of the image. However, this method results in similar search results for two electromagnetic signal images that are visually very similar but have different contents, which often results in wrong data being searched, greatly reducing the accuracy of the search.

[0004] Therefore, how to improve the accuracy of electromagnetic signal image search has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The embodiments of the present application provide a method, device, equipment, medium and product for searching electromagnetic signal images, which can solve the problem of how to improve the accuracy of electromagnetic signal image searching.

[0006] In a first aspect, an embodiment of the present application provides a method for searching electromagnetic signal graphics data, comprising: The image-text generation model is used to obtain the target electromagnetic signal image and target text content corresponding to the data to be searched; Extracting image semantic features of a target electromagnetic signal image and text semantic features of target text content, wherein the image semantic features indicate electromagnetic signals in the target electromagnetic signal image, and the text semantic features indicate at least one of a spectrum feature, a signal mode, or a propagation environment of the electromagnetic signal; Fuse the image semantic features and text semantic features to obtain the target image and text semantic features; Searching for a target reference image-text semantic feature corresponding to the target image-text semantic feature from a search graph corresponding to a knowledge base, wherein the knowledge base includes a plurality of reference electromagnetic signal images and a plurality of reference text contents, and the search graph includes a plurality of reference image-text semantic features, and one reference image-text semantic feature corresponds to at least one reference electromagnetic signal image or at least one reference text content; In the knowledge base, the reference electromagnetic signal image or reference text content corresponding to the target reference image and text semantic features is determined as the search result of the data to be searched.

[0007] In some embodiments, the search graph is composed of multiple groups of nodes, each group of nodes includes a central node, and the search graph corresponding to the knowledge base is searched for the target reference image-text semantic feature corresponding to the target image-text semantic feature, including: Determine the degree of membership between the reference image-text semantic features and the target image-text semantic features corresponding to each central node; Determine the central node corresponding to the membership degree that meets the first condition as the target node; Taking the target node as the search starting point, searching at least one node in the search graph until the stop condition is met, and obtaining the vector similarity between the reference image-text semantic feature corresponding to at least one node and the target image-text semantic feature; Based on the vector similarity and the reference image-text semantic feature corresponding to at least one node, a target reference image-text semantic feature is determined.

[0008] In some embodiments, extracting image semantic features of the target electromagnetic signal image and text semantic features of the target text content includes: Extract image semantic features from the target electromagnetic signal image using an image feature extraction model; The text feature extraction model is used to extract text semantic features from the target text content.

[0009] In some embodiments, the image feature extraction model includes a convolutional layer, a temporal layer, a spatial layer, and a fully connected layer; The convolution layer is used to extract local features of the target electromagnetic signal image and input the local features into the temporal layer and the spatial layer, where the local features indicate the electromagnetic signal; The time series layer is used to extract time series features from local features and input them into the fully connected layer. The time series features indicate the trend of electromagnetic signals changing over time. The spatial layer is used to extract spatial features from local features and input the spatial features into the fully connected layer. The spatial features include the trend of electromagnetic signals changing with space. The fully connected layer is used to perform weighted fusion of temporal features and spatial features to obtain image semantic features and output image semantic features.

[0010] In some embodiments, the image semantic features and the text semantic features are fused to obtain the image and text semantic features, including: Perform feature alignment operations on the image semantic features and the text semantic features respectively to obtain new image semantic features and new text semantic features; Based on the text weight and image weight, the new image semantic features and the new text semantic features are weightedly fused to obtain the target image and text semantic features.

[0011] In some embodiments, the image-text generation model is used to obtain the target electromagnetic signal image and target text content corresponding to the data to be searched, including: In the case where the data to be searched is an electromagnetic signal image, a target text content corresponding to the data to be searched is generated using a picture-text generation model, and the electromagnetic signal image is used as a target electromagnetic signal image; In the case where the data to be searched is text data, an electromagnetic signal image corresponding to the text data is generated using the image-text generation model, the generated electromagnetic signal image is used as the target electromagnetic signal image, and the text data is used as the target text content; When the data to be searched includes a first text material and a first electromagnetic signal image, a graphic-text generation model is used to generate a second electromagnetic signal image corresponding to the first text material, and to generate a first text content corresponding to the first electromagnetic signal image. The first electromagnetic signal image and the second electromagnetic signal image are used as target electromagnetic signal images, and the first text material and the first text content are used as target text content.

[0012] In a second aspect, an embodiment of the present application provides a data search device for electromagnetic signal graphics, including: An acquisition module, used to acquire a target electromagnetic signal image and target text content corresponding to the data to be searched by using an image-text generation model; An extraction module, used to extract image semantic features of a target electromagnetic signal image and text semantic features of a target text content, wherein the image semantic features indicate electromagnetic signals in the target electromagnetic signal image, and the text semantic features indicate at least one of a spectrum feature, a signal mode, or a propagation environment of the electromagnetic signal; A fusion module is used to fuse image semantic features and text semantic features to obtain target image and text semantic features; A search module is used to search for a target reference image-text semantic feature corresponding to a target image-text semantic feature from a search graph corresponding to a knowledge base, wherein the knowledge base includes a plurality of reference electromagnetic signal images and a plurality of reference text contents, and the search graph includes a plurality of reference image-text semantic features, wherein one reference image-text semantic feature corresponds to one reference electromagnetic signal image or one reference text content; The determination module is used to determine the reference electromagnetic signal image or reference text content corresponding to the target reference image and text semantic features in the knowledge base as the search result of the data to be searched.

[0013] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the electronic device implements the data search method for electromagnetic signal graphics as described in any embodiment of the first aspect.

[0014] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and wherein when the computer program is executed by a processor, the method for searching for electromagnetic signal graphics data as described in any embodiment of the first aspect is implemented.

[0015] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the electromagnetic signal image data search method described in any embodiment of the first aspect above.

[0016] Compared with the prior art, the beneficial effects of the embodiments of the present application are as follows: the target electromagnetic signal image and target text content corresponding to the data to be searched are obtained by using a graph-text generation model, and image semantic features and text semantic features are extracted and fused from the target electromagnetic signal image and target text content respectively, to obtain target graph-text semantic features that can comprehensively represent the electromagnetic signal image semantic content and text semantic content corresponding to the data to be searched. When searching, the target reference graph-text semantic features corresponding to the target graph-text semantic features are searched from the retrieval graph corresponding to the knowledge base, so as to determine the reference electromagnetic signal image or reference text content corresponding to the target reference graph-text semantic features in the knowledge base as the search result. Searching is performed using the semantic features corresponding to the data to be searched, taking into account the understanding of the semantic content, avoiding searching through basic visual features in the electromagnetic signal image, and improving the search accuracy. In addition, the reference image and text semantic features correspond to at least one reference electromagnetic signal image or reference text content in the knowledge base, thereby realizing a matching search between multiple semantic features at the electromagnetic signal image or text content level, avoiding the extraction of multiple visual features of the target electromagnetic signal image, and matching each visual feature of the target electromagnetic signal image with each visual feature of the reference electromagnetic signal image, thereby greatly reducing the amount of calculation and improving the retrieval efficiency.

[0017] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 It is a schematic diagram of a flow chart of a data search method for electromagnetic signal graphics provided by an embodiment of the present application; Figure 2It is a workflow diagram of a data search method for electromagnetic signal graphics in an application scenario provided by an embodiment of the present application; Figure 3 It is a search flow chart of a data search method for electromagnetic signal graphics in an application scenario provided by an embodiment of the present application; Figure 4 It is a flow chart of another method for searching electromagnetic signal images and texts provided in an embodiment of the present application; Figure 5 It is a structural schematic diagram of the image extraction feature model provided in the embodiment of the present application; Figure 6a It is a schematic diagram of the structure of the BERT model provided in the embodiment of the present application; Figure 6b It is a structural diagram of a text feature extraction model provided in an embodiment of the present application; Figure 7 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application; Figure 8 It is a structural schematic diagram of a data search device for electromagnetic signal graphics provided in an embodiment of the present application. DETAILED DESCRIPTION

[0020] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0021] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0022] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0023] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0024] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0025] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0026] In the field of graph-based knowledge management, current electromagnetic signal image search technologies are mostly based on visual features, and semantic understanding is still limited. Even if two images are very similar visually, their semantic content may be completely different, but current electromagnetic signal image search technologies cannot handle such scenarios. In other words, current electromagnetic signal image search technologies are mainly based on matching based on low-level visual features (such as color, texture, shape), and it is difficult to understand the high-level semantic information of images (such as scenes, emotions, events, etc.). There are challenges in multi-object recognition and semantic understanding in complex scenes.

[0027] In addition, for complex scenes, multiple objects or partially occluded images, the current electromagnetic signal image search technology cannot effectively process and identify them, and usually relies on a large number of annotations to improve the accuracy of retrieval, but the quality and completeness of the annotations have a great impact on the retrieval results. Moreover, based on the annotations, the current electromagnetic signal image search technology needs to extract and match the complex visual features of the electromagnetic signal image to obtain the search results, which requires huge calculations and low search efficiency.

[0028] In response to the above technical problems, the embodiments of the present application provide a method, device, equipment, medium and product for searching electromagnetic signal images and text data. In the search scenario, the target electromagnetic signal image and target text content corresponding to the data to be searched are obtained, and the image semantic features and text semantic features are extracted and fused from the target electromagnetic signal image and the target text content respectively to obtain the target image and text semantic features corresponding to the data to be searched. In the retrieval graph, multiple reference image and text semantic features are searched based on the target image and text semantic features, and the reference image and text semantic features correspond to at least one reference electromagnetic signal image or reference text content in the knowledge base, so as to achieve matching search between multiple semantic features at the electromagnetic signal image or text content level, realize understanding of semantic content during search and improve search accuracy, avoid extraction of multiple visual features of the target electromagnetic signal image, and match each visual feature of the target electromagnetic signal image with each visual feature of the reference electromagnetic signal image, greatly reduce the amount of calculation and improve retrieval efficiency.

[0029] The following describes the data search method for electromagnetic signal graphics in the embodiment of the present application in conjunction with the accompanying drawings.

[0030] Figure 1 is a flow chart of a method for searching electromagnetic signal images and texts according to an embodiment of the present application. Figure 1 The method shown comprises the following steps: Step S101, using the image-text generation model to obtain the target electromagnetic signal image and target text content corresponding to the data to be searched.

[0031] The data search method for electromagnetic signal graphics provided in the embodiments of the present application can be applied to electronic devices such as mobile phones, tablet computers, desktop computers, laptop computers, ultra-mobile personal computers (ultra-mobile personal computers, UMPC), netbooks, personal digital assistants (personal digital assistants, PDAs) or servers. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network, content distribution network), and big data and artificial intelligence platforms. It can also be a special or platform server that provides Internet of Vehicles services, road network collaboration, vehicle-road collaboration, intelligent transportation, automatic driving, industrial Internet services, data communications (such as 4G, 5G, etc.). The embodiments of the present application do not impose any restrictions on the specific types of electronic devices.

[0032] In this embodiment, the target text content indicates electromagnetic signal characteristic information such as the spectrum characteristics, signal mode or propagation environment of the electromagnetic signal in the target electromagnetic signal image. The image-text conversion model supports image-text conversion for matching electromagnetic signal images and texts, and can convert electromagnetic signal images into corresponding text contents and convert text contents into corresponding electromagnetic signal images, that is, mapping electromagnetic signal images and text contents into a unified vector space.

[0033] The image-text generation model can be an open source multimodal large model that has been fine-tuned using a specific data set. The open source multimodal large model can be omnilmm, mm-interleaved, viscpm, or an open source multimodal large model based on qwen-vl, etc. The embodiment of the present application takes an open source multimodal large model based on qwen-vl as an example. The specific data set includes multiple positive and negative sample pairs, the positive sample pairs include mutually corresponding sample electromagnetic signal images and sample text content, and the negative sample pairs include mutually unrelated sample electromagnetic signal images and sample text content. Based on the specific data set prepared in advance (ensuring that the samples are diverse and representative), the open source multimodal large model is pre-trained or fine-tuned by contrastive learning, so that the model can directly calculate the similarity between the electromagnetic signal image and the text content in the vector space, thereby obtaining a picture-text conversion model. During training, the multimodal model can automatically adjust the similarity calculation strategy of the sample electromagnetic signal image and the sample text content according to the context and application requirements, and continuously optimize the similarity adjustment algorithm. By maximizing the similarity between positive samples (corresponding sample electromagnetic signal images and sample text contents) and minimizing the similarity between negative samples (unrelated sample electromagnetic signal images and sample text contents), an effective feature representation is learned to obtain an image-text conversion model.

[0034] The user can input the electromagnetic signal image or text content as the data to be searched into the electronic device, and the electronic device inputs the received data to be searched into the image-text conversion model. The image-text conversion module can convert the electromagnetic signal image into text content or convert the text content into the electromagnetic signal image and output it, so that the electronic device obtains the target electromagnetic signal image and target text content.

[0035] In one implementation, the data to be searched includes an electromagnetic signal image or text material, and a target electromagnetic signal image and a target text content corresponding to the data to be searched are obtained by using a graph-text generation model, including: when the data to be searched is an electromagnetic signal image, the target text content corresponding to the data to be searched is generated by using the graph-text generation model, and the electromagnetic signal image is used as the target electromagnetic signal image; when the data to be searched is text material, the electromagnetic signal image corresponding to the text material is generated by using the graph-text generation model, and the generated electromagnetic signal image is used as the target electromagnetic signal image, and the text material is used as the target text content; when the data to be searched includes first text material and a first electromagnetic signal image, the graph-text generation model is used to generate a second electromagnetic signal image corresponding to the first text material, and a first text content corresponding to the first electromagnetic signal image is generated, the first electromagnetic signal image and the second electromagnetic signal image are used as target electromagnetic signal images, and the first text material is used as well as the first text content as the target text content.

[0036] See also Figure 2 as well as Figure 3 When the data to be searched is an electromagnetic signal image, the electronic device enters the image-to-text stage, and inputs the electromagnetic signal image into the image-to-text conversion model. The image-to-text conversion model can generate and output a text description corresponding to the electromagnetic signal image. The text description is the target text content, and the electromagnetic signal image is used as the target electromagnetic signal image. When the data to be searched is text data, the electronic device enters the text-to-image stage, and inputs the text data into the image-to-text conversion model. The image-to-text conversion model will perform preprocessing operations on the text data to understand the intention of the text data, and then draw a waveform electromagnetic signal image containing features such as pulses, spectra, carrier frequencies and / or pulse widths according to time changes, and output the waveform electromagnetic signal image as the target electromagnetic signal image. The text data is used as text content. The preprocessing operation includes eliminating abnormal data, analyzing grammar or understanding intentions. In this way, the target electromagnetic signal image and target text content corresponding to the data to be searched can be quickly obtained through the image-to-text conversion model, providing data support for subsequent searches.

[0037] Step S102: extracting image semantic features of the target electromagnetic signal image and text semantic features of the target text content.

[0038] In this embodiment, the image semantic feature indicates the electromagnetic signal in the target electromagnetic signal image, and the text semantic feature indicates at least one of the spectrum characteristics, signal mode or propagation environment of the electromagnetic signal. An electromagnetic signal is an electromagnetic signal. Electromagnetic signal images mainly include two forms, one is an electromagnetic signal image that changes with time, and the other is an electromagnetic signal image that changes with coil displacement (that is, with space). In addition, the direction of change of the magnetic signal is also an important dimension for analyzing electromagnetic information. When searching for the target electromagnetic signal image, if the trend of the electromagnetic signal in the target electromagnetic signal image changing with time and space (that is, the image semantic feature of the target electromagnetic signal image) can be extracted, the target electromagnetic signal image can be accurately analyzed to improve the accuracy of the search results.

[0039] The electronic device can extract the image semantic features of the target electromagnetic signal image through a neural network. The neural network can be a convolutional neural network, a recurrent neural network or an autoencoder, etc., and the text semantic features of the target text content are extracted through natural language processing (Natural Language Processing, NLP) technology. The specific implementation method of the electronic device extracting image semantic features and text semantic features will be introduced in subsequent embodiments.

[0040] Step S103, fusing the image semantic features and the text semantic features to obtain the target image and text semantic features.

[0041] The electronic device can align the image semantic features and the text semantic features and perform weighted fusion, and the fused semantic features are used as the target image and text semantic features.

[0042] Step S104, searching the target reference image-text semantic feature corresponding to the target image-text semantic feature from the search graph corresponding to the knowledge base.

[0043] In this embodiment, the knowledge base includes multiple reference electromagnetic signal images and multiple reference text contents, and the retrieval graph includes multiple reference image and text semantic features, and one reference image and text semantic feature corresponds to at least one reference electromagnetic signal image or at least one reference text content. In the retrieval graph, multiple reference image and text semantic features are each a node, and the nodes of the multiple reference image and text semantic features include a first node and a second node, and the first node is a neighbor node of the second node. The target image and text semantic features have the same dimension as the reference image and text semantic features in the retrieval graph.

[0044] The electronic device will select a node from the search graph as the search starting point. It can randomly select a node as the starting point, or select a specific node based on some prior knowledge or specific rules. For example, a representative node in the area related to the semantic features of the target image and text is selected as the starting point.

[0045] The electronic device determines the vector similarity between the target image-text semantic feature and the reference image-text semantic feature corresponding to the node from the node corresponding to the search starting point. The vector similarity can be the Euclidean distance or cosine similarity between the two features. And based on the connection relationship of the node in the search map, the neighbor node of the node is obtained. The neighbor node is a node directly connected to the node, and the neighbor node has a certain association with the node in the search map. For each neighbor node, the electronic device still determines the vector similarity between the target image-text semantic feature and the reference image-text semantic feature corresponding to the neighbor node. And based on the determined vector similarity, the candidate node for the next search is selected from all neighbor nodes. The specific screening condition can select the neighbor node corresponding to the reference image-text semantic feature corresponding to the vector similarity less than or equal to the preset threshold, or select the neighbor node corresponding to the reference image-text semantic feature corresponding to the highest vector similarity. The specific screening condition can be based on the self-setting embodiment of the present application without specific restrictions.

[0046] Repeat the above process of obtaining neighbor nodes and selecting candidate nodes based on the candidate nodes until the search stop condition is met, sort the vector similarities between the target image and text pre-extraction feature determined in the search process and at least one reference image and text semantic feature in order from high to low, and determine the reference image and text semantic features corresponding to the vector similarities ranked in the top X positions in the sorting result as the target reference image and text semantic features, where X is a positive integer, which can be determined based on the search conditions input by the user to the electronic device. For example, when the user outputs the data to be searched to the electronic device, the user simultaneously inputs the search condition of "give an example of 3 similar electromagnetic signal images", then X is 3, or inputs "please explain the signal mode of the electromagnetic signal in the electromagnetic signal image", then X is 1. The stop condition can be that the maximum search depth is reached, the vector similarity meets the requirements (for example, a node is found, and the vector similarity between the corresponding reference image and text semantic features and the target image and text semantic features reaches the threshold, or a sufficiently similar node has been found on the current search path) or the search range is exhausted, which can be set by the user and is not specifically limited in the embodiment of the present application.

[0047] Step S105 , in the knowledge base, the reference electromagnetic signal image or reference text content corresponding to the target reference image and text semantic feature is determined as the search result of the data to be searched.

[0048] Each reference image and text semantic feature in the retrieval map is associated with the electromagnetic signal image identification information of the corresponding reference electromagnetic signal image or the text identification information of the reference text content. After obtaining the target reference image and text semantic feature, the electronic device can search for the reference electromagnetic signal image corresponding to the electromagnetic signal image identification information or the reference text content corresponding to the text identification information from the knowledge base based on the electromagnetic signal image identification information or text identification information corresponding to the reference image and text semantic feature, thereby obtaining search results.

[0049] In the embodiment of the present application, the target electromagnetic signal image and target text content corresponding to the data to be searched are obtained by using the image-text generation model, and the image semantic features and text semantic features are extracted and fused from the target electromagnetic signal image and the target text content respectively, so as to obtain the target image-text semantic features that can comprehensively represent the electromagnetic signal image semantic content and text semantic content corresponding to the data to be searched. During the search, the target reference image-text semantic features corresponding to the target image-text semantic features are searched from the retrieval graph corresponding to the knowledge base, so as to determine the reference electromagnetic signal image or reference text content corresponding to the target reference image-text semantic features in the knowledge base as the search result. The search is realized by the semantic features corresponding to the data to be searched, the understanding of the semantic content is taken into account, the search is avoided by the basic visual features in the electromagnetic signal image, and the search accuracy is improved. In addition, the reference image-text semantic features correspond to at least one reference electromagnetic signal image or reference text content in the knowledge base, and the matching search between multiple semantic features at the electromagnetic signal image or text content level is realized, the extraction of multiple visual features of the target electromagnetic signal image is avoided, and each visual feature of the target electromagnetic signal image is matched with each visual feature of the reference electromagnetic signal image, which greatly reduces the amount of calculation and improves the retrieval efficiency.

[0050] Figure 4 FIG. 1 is a flow chart of another method for searching electromagnetic signal images and texts according to an embodiment of the present application. Figure 4 The method shown comprises the following steps: Step S201, using the image-text generation model to obtain the target electromagnetic signal image and target text content corresponding to the data to be searched. For details, see Figure 1 Step S101 in the illustrated embodiment is not described in detail here.

[0051] In one implementation, extracting the image semantic features of the target electromagnetic signal image and the text semantic features of the target text content includes the following steps S202 to S203: Step S202: extracting image semantic features from the target electromagnetic signal image using an image feature extraction model.

[0052] The image feature extraction model (hereinafter referred to as the IME-Net model) is a pre-trained convolutional neural network that can extract the temporal and spatial trends of electromagnetic signals in electromagnetic signal images. After the electronic device inputs the target electromagnetic signal image into the image feature extraction model, the image semantic features output by the image feature extraction model can be obtained.

[0053] In one implementation, the image feature extraction model includes a convolutional layer, a temporal layer, a spatial layer, and a fully connected layer; The convolution layer is used to extract local features of the target electromagnetic signal image and input the local features into the temporal layer and the spatial layer, where the local features indicate the electromagnetic signal; The time series layer is used to extract time series features from local features and input the time series features into the fully connected layer. The time series features indicate the trend of electromagnetic signals changing over time. The spatial layer is used to extract spatial features from local features and input the spatial features into the fully connected layer. The spatial features include the trend of electromagnetic signals changing with space. The fully connected layer is used to perform weighted fusion of temporal features and spatial features to obtain image semantic features and output image semantic features.

[0054] In the above technical scheme, after the local features of the electromagnetic signal image are extracted through the convolution layer, the indicative temporal features and spatial features are respectively extracted from the local features through a special temporal layer and a spatial layer to accurately capture the changes of the electromagnetic signal in space and time, respectively. Then, the temporal features and the spatial features are weighted and fused into image semantic features through a fully connected layer, and the trends of the electromagnetic signal changes with space and with time are fused together, so that the image semantic features include the comprehensive change trends of the electromagnetic signal in time and space, providing reliable semantic content in the electromagnetic signal image for the search. In subsequent searches, the trends of the electromagnetic signal changes with time and space reflected in the target electromagnetic signal image are also taken into account, thereby improving the accuracy of the search.

[0055] In one implementation, the specific structure of the image extraction feature model can be as follows: Figure 5 As shown in the figure, the IME-Net model consists of a layer of convolutional neural network (CNN), 2 layers of dilated convolutions (DCN), 4 layers of temporal convolutional network (TCN) and a layer of fully connected network, where CNN is the convolution layer, 2 layers of DCN are the spatial layer, 4 layers of TCN are the temporal layer, and the fully connected network is the fully connected layer. The DCN network can control the network's receptive range by adjusting the expansion rate d of the convolution, and can effectively learn the feature information of different spatial scales of the image. The TCN network is a variant of the causal convolutional network, which can effectively mine the sequence features in the feature information and is usually used for processing time series data.

[0056] Specifically, the electronic device uses a layer of convolutional neural network (CNN) through the IME-Net model to extract local features of the target electromagnetic signal image input into the model. , and then use the 2-layer DCN network to learn (or extract) the local features of the target electromagnetic signal image ( ) Spatial characteristics at different spatial scales , a 4-layer TCN network is used to extract the local features of the target electromagnetic signal image ( ) Time series characteristics that change over time Finally, a fully connected network with RELU activation function is used to correct the temporal features and spatial features to avoid gradient disappearance in the subsequent search process, and the temporal features and spatial features are weightedly fused to obtain the image semantic features. , the fusion method is shown in formula (1), formula (1) is .

[0057] in, , represents the fusion weight corresponding to the spatial feature, represents the fusion weight corresponding to the temporal feature, Represents spatial features, Represents temporal features. The above image feature extraction model can effectively extract the spatial features and temporal features of electromagnetic signal images. By adjusting the network parameters, the algorithm's ability to capture global features can also be expanded.

[0058] Step S203: extracting text semantic features from the target text content using a text feature extraction model.

[0059] The text feature extraction model (hereinafter referred to as the WE-Net model) is a pre-trained natural semantic processing model that can extract contextual semantic information from text content. After the electronic device inputs the target text content into the text feature extraction model, it can obtain the semantic features of the model output text.

[0060] In one implementation, the WE-Net model is a modified Bidirectional Encoder Representations from Transformers (BERT) model. The BERT model can effectively capture the contextual information around the target word from the input sequence, enabling the model to better understand the meaning of the sentence.

[0061] like Figure 6aAs shown in the figure, the structure of the BERT model before the improvement includes word vector embedding layers, encoders (Transformer Encoders), and pooling output layers. The word vector embedding layer is used to convert each word in the text content into a vector representation of a fixed dimension to obtain a word vector. The encoder is used to perform deep semantic feature extraction and encoding on the word vector. The pooling output layer is used to perform pooling operations on the semantic features output by the encoder to obtain a text representation of a fixed length. The BERT model can output results in different forms according to specific task requirements. In some tasks, such as text generation, sequence output can be performed, and the sequence output can be a text sequence generated word by word; in classification tasks, classification output can be performed, and the classification output can be used to determine the specific category to which the input text belongs.

[0062] like Figure 6b As shown, in the embodiment of the present application, a fully connected layer is added on the basis of the BERT model, so that after the semantic features of the text content are obtained, the semantic features are corrected through the fully connected layer using the activation function RELU, so that the text semantic features and image semantic features obtained based on the reference electromagnetic signal image or the reference text content are integrated to construct a retrieval map, while improving the reliability of the text semantic features of the target text content during the search process.

[0063] In the technical solution described in step S202 to step S203, different types of semantic features (image semantic features of the target electromagnetic signal image and text semantic features of the target text content) are extracted by the image feature extraction model and the text feature extraction model respectively, and different types of semantic features are allocated to professional models for extraction, which can avoid using the same model to extract inaccurate or incomplete semantic features, improve the reliability of image semantic features and text semantic features, and thus improve the accuracy of subsequent retrieval. In addition, different models can extract representative semantic features from electromagnetic signal images and text content respectively, and these semantic features can retain richer information after subsequent fusion. In the search scenario, electromagnetic signal images and text content can be searched based on the fused semantic features, realizing cross-modal search while determining the accuracy of search results.

[0064] In one implementation, the image semantic features and the text semantic features are fused to obtain the target image and text semantic features, including the following steps S204 to S205: Step S204: performing feature alignment operations on the image semantic features and the text semantic features respectively to obtain new image semantic features and new text semantic features.

[0065] The electronic device can map the image semantic features into a vector space through the electromagnetic signal image vector mapping relationship, and map the text semantic features into the same vector space through the text vector mapping relationship, thereby completing the feature alignment operation and obtaining the mapped image semantic features and text semantic features, that is, the new image semantic features and the new text semantic features.

[0066] In one implementation, the electromagnetic signal image vector mapping relationship and the text vector mapping relationship can be respectively expressed as formula (2) and formula (3). Formula (2) is: , formula (3) is .

[0067] in, Represents the semantic features of new text, Represents the semantic features of the image, Represents the semantic features of the image, Represents the semantic features of text, represents the alignment parameters corresponding to the semantic features of the image, Indicates the alignment parameters corresponding to the text semantic features, represents the alignment weight corresponding to the semantic features of the image, Represents the alignment weight corresponding to the text semantic features.

[0068] In one implementation, the electronic device uses a graphic feature extraction model to perform feature alignment operations on image semantic features and text semantic features respectively to obtain new image semantic features and new text semantic features. The graphic feature extraction model includes an image feature extraction model, a text feature extraction model and a graphic generation model.

[0069] Step S205 , based on the text weight and the image weight, weighted fusion is performed on the new image semantic features and the new text semantic features to obtain the target image-text semantic features.

[0070] The electronic device can determine a first product result between the text weight and the new text semantic feature, and determine a second product result between the image weight and the new image semantic feature, and determine the sum of the first product result and the second product result as the target image and text semantic feature.

[0071] In one implementation, the electronic device can determine the semantic features of the image and text by using the following formula (4), where: ,in, F Represents the semantic features of images and texts. represents the image weight, A Represents the text weight.

[0072] In one implementation, the electronic device determines the semantic features of the target image and text through the above formulas (2) to (4) through the cross-attention learning mechanism. At this time, the electronic device first determines the text weight in this weighted fusion based on the new image semantic features and the new text semantic features. Specifically, the text weight is determined based on the following formulas (5) to (7). Formula (5) is , formula (6) is , formula (7) is .

[0073] in, Q represents the query vector, K represents a key-value vector, represents the query weight, represents the key value weight, softmax represents the normalization function, express K Dimension.

[0074] In one implementation, the electronic device uses a feature fusion module in a graphic feature extraction model to perform weighted fusion on new image semantic features and new text semantic features based on text weights and image weights to obtain target graphic semantic features.

[0075] Specifically, when the electronic device can use the aforementioned specific data set and the cross-attention mechanism to train the image-text feature extraction model to fuse the sample image semantic features and the sample text semantic features corresponding to the sample electromagnetic signal image, the above formulas (2) to (7) involve , , , and And other parameters.

[0076] See also Figure 2The workflow diagram of data search in an application scenario shown in , when training the image-text feature extraction model, the electronic device can divide a specific data set into a training set and a test set, and input the training set into the image-text feature extraction model, wherein the image-text conversion model generates a first text content corresponding to the first sample electromagnetic signal image in the training set, and outputs the text content to the text feature extraction model, and outputs the first sample electromagnetic signal image to the image feature extraction model. The image feature extraction model in the image-text feature extraction model extracts the first image semantic feature of the first sample electromagnetic signal image, and the text feature extraction model in the image-text feature extraction model extracts the first text semantic feature of the first text content, and then enters the feature fusion stage, the first sample electromagnetic signal image is any sample electromagnetic signal image in the training set, and the sample text content in the training set will also enter the feature fusion stage through a similar processing flow, except that the image-text conversion model generates an electromagnetic signal image corresponding to the sample text content, and inputs the electromagnetic signal image and the sample text content into the image feature extraction model and the text feature extraction model, respectively.

[0077] In the feature fusion stage of the training process, the image semantic features obtained in the above process are assumed to be , the text semantic features are The electronic device uses the feature fusion module in the image-text feature extraction model to perform feature alignment based on formula (2) and formula (3) to obtain the semantic features of the new image. And the semantic features of the new text ,in, , , as well as are learnable parameters.

[0078] Then, based on formula (5) and formula (6), we can determine Q and K ,in, and is a learnable parameter. Based on formula (7), the attention weight, i.e., the text weight, is determined. Then, the text weight and the image weight are used to and Weighted fusion can be performed to form semantic features of images and texts F .

[0079] In the training stage, considering that the subsequent data retrieval process is mainly used to compare the similarity between the semantic features of the target image and text and the semantic features of the reference image and text in the retrieval spectrum, the contrast loss function Triplet Loss can be used to define the loss function. ,in, D is the Euclidean distance, a is the anchor feature, pis the positive sample feature (the semantic feature of the sample image and text corresponding to the positive sample pair), n is the negative sample feature (the semantic feature of the sample image and text corresponding to the negative sample pair), margin is the preset boundary value. Based on the training set, the gradient descent method is used to minimize the loss function and update , , , and The electronic device obtains the parameters involved in the above formulas (2) to (7). Using the trained parameters, the semantic features of the target image and text can be obtained based on the above formulas (2) to (7).

[0080] In the technical solution of steps S204 to S205, feature alignment operations are performed on image semantic features and text semantic features respectively, so that features of two different dimensions can be mapped to the same vector space, and new image semantic features and new text semantic features in the same vector representation form can be obtained, so as to facilitate weighted fusion of the new image semantic features and new text semantic features based on text weights and image weights, and ensure the interpretability and comprehensiveness of the information in the image semantic features and text semantic features. Weighted fusion of the new image semantic features and new text semantic features can make the fused features include information in both the electromagnetic signal image and the text content, thereby providing a reliable data foundation for subsequent searches.

[0081] In one implementation, searching for a target reference image-text semantic feature corresponding to the target image-text semantic feature from a search graph corresponding to a knowledge base includes the following steps S206 to S209; Step S206 , determining the degree of membership between the reference image-text semantic features corresponding to each central node and the target image-text semantic features.

[0082] The search graph is composed of multiple groups of nodes, each of which includes a central node, which is also the representative node of a group of nodes.

[0083] See also Figure 3 In the application scenario shown in the figure, for each central node, the electronic device determines the characteristic distance between the reference image and text semantic features corresponding to the central node and the target image and text semantic features, and determines the natural logarithm e and the negative power of the characteristic distance as the membership degree between the reference image and text semantic features corresponding to the central node and the target image and text semantic features. Specifically, the characteristic distance between the reference image and text semantic features corresponding to each central node and the target image and text semantic features can be substituted into the formula Determine the membership degree corresponding to each central node, where represents the feature distance between the target image-text semantic feature and the reference image-text semantic feature corresponding to the i-th central node, Indicates the degree of membership. The feature distance can be the Euclidean distance or the distance between feature vectors determined by other methods.

[0084] In one implementation, the electronic device determines the feature distance between the reference image-text semantic feature corresponding to the central node and the target image-text semantic feature by using the following formula (8), where: .

[0085] in, Representing the semantic features of images and texts Semantic features of images and texts The characteristic distance between The operation represents the subtraction of the vector (graphic and text semantic features) element points and takes the absolute value of the difference. It can be seen from formula (8) that when the difference between the semantic features of two images in different directions is smaller, the distance d will be smaller, and vice versa d The bigger it will be.

[0086] It should be noted that before searching in the search graph, it is necessary to construct a search graph corresponding to the knowledge base. In one implementation, before searching for the target reference image semantic features corresponding to the target image semantic features from the search graph corresponding to the knowledge base, the electromagnetic signal image data search method further includes the following steps (1) to (3): Step (1) obtains reference image-text semantic features corresponding to each reference electromagnetic signal image and each reference text content, thereby obtaining a plurality of reference image-text semantic features.

[0087] The specific process of the electronic device obtaining the reference image and text semantic features can be found in the process of obtaining the target image and text semantic features, which will not be described in detail here.

[0088] Step (2) constructs an initial search graph based on multiple reference image and text semantic features. The initial search graph consists of multiple nodes, and each node corresponds to a reference image and text semantic feature.

[0089] Electronic devices can build an initial retrieval graph based on the Neighborhood Structure Graph algorithm (NSG) in the Approximate Nearest Neighbors Search (ANNS) algorithm based on multiple reference graph semantic features. The NSG algorithm optimizes the graph node index construction algorithm to reduce the node degree to improve search efficiency. The core idea of ​​the NSG algorithm is that not all neighboring points contribute to search efficiency. Therefore, in the process of building the initial retrieval graph, if multiple neighbors of point P are crowded together, only one point needs to be selected from these neighbors for search, and other points may be redundant.

[0090] The reference image semantic features obtained after feature fusion already include the image semantic features of the reference electromagnetic signal image and the contextual high-level semantic features (i.e., text semantic features). Therefore, when constructing the initial retrieval graph based on the NSG algorithm, it is necessary to quantify the differences in the semantic features of different reference images and texts to obtain the index distance between the graph nodes, i.e., the feature distance. The calculation of the feature distance is shown in formula (8). This feature distance is used as the basis for constructing the initial retrieval graph.

[0091] Step (3), in the initial search graph, clustering operations are performed on the multiple nodes according to the semantic features of the reference images and texts corresponding to the multiple nodes to obtain multiple groups of nodes and multiple central nodes. The clustering operation includes determining the central node and classifying the multiple nodes.

[0092] Considering that traditional retrieval graph-based searches have a fixed retrieval starting point (enter point), if the semantic features of the target reference image and text being searched are far away from the retrieval starting point, many unnecessary nodes will be traversed. Although the NSG algorithm prunes the edges in the process of constructing the initial retrieval graph, the search efficiency is not significantly improved.

[0093] In order to further improve the search efficiency, a coarse search layer can be constructed on the basis of the initial search graph (that is, multiple nodes in the initial search graph are divided into multiple groups, and a central node is set for each group, and the central node can be regarded as the search starting point among multiple nodes). In order to avoid the problem of memory space explosion like the traditional Hierarchical Navigable Small World algorithm (HNSW algorithm), the K-means clustering (K-Means) algorithm is used to cluster the semantic features of the reference images and texts corresponding to multiple nodes, and obtain K clusters (that is, K groups of nodes) and the corresponding cluster centers. (That is, the central nodes corresponding to multiple groups of nodes are obtained), each cluster center can be regarded as K search entries of the Base Graph, and the search graph is obtained. In subsequent searches, the cluster center (that is, the target central node) closest to the calculated target image semantic feature data is first found in the coarse search layer (that is, among multiple central nodes), and then the search is performed in the search graph with this cluster center as the search starting point, and the union of the search results (all searched target reference image semantic features) is returned.

[0094] If the number of nodes in the search graph is too large, the number of coarse search layers can be increased to improve the search speed. Although filtering the coarse search layer will reduce the search accuracy to a certain extent, the error is controllable if the K-Means algorithm is used correctly and the number of coarse search layers is controlled.

[0095] Step S207: determine the central node corresponding to the membership degree that meets the first condition as the target node.

[0096] The first condition may be the membership with the highest value among all the memberships. The electronic device selects the membership with the highest value from all the obtained memberships and determines the central node corresponding to the selected membership as the target node.

[0097] Step S208, taking the target node as the search starting point, searching at least one node in the search graph until the stop condition is met, and obtaining the vector similarity between the reference image-text semantic feature corresponding to at least one node and the target image-text semantic feature.

[0098] The electronic device will search the search spectrum with the target node as the search starting point until the stop condition is reached. The specific search process is the same as Figure 1 The search process in step S104 in the illustrated embodiment is similar and will not be elaborated on again. For each node electronic device searched, the electronic device will determine the vector similarity between the reference graphic semantic feature corresponding to the node and the target graphic semantic feature, thereby obtaining the vector similarity between the reference graphic semantic feature corresponding to at least one node and the target graphic semantic feature.

[0099] Step S209 : determining a target reference image-text semantic feature based on the vector similarity and the reference image-text semantic feature corresponding to at least one node.

[0100] The electronic device may sort each obtained vector similarity in order from high to low, and determine the reference image-text semantic features corresponding to the vector similarities ranked in the top X positions in the sorting result as the target reference image-text semantic features.

[0101] In the technical scheme of step S206 to step S209, the degree of membership between the target image semantic features and the reference image semantic features corresponding to the central nodes of each group of nodes in the search graph is firstly used to screen out the central nodes corresponding to the degree of membership that meets the first condition, and the target image semantic features are searched with this central node as the search starting point. In this way, it is possible to perform a rough search of the search graph first by determining the search starting point, and then perform a detailed search of the search graph from the search starting point, perform regional search on multiple nodes in the search graph, avoid the traditional situation of searching the entire graph from a specific node, improve the search efficiency of the search graph, and then improve the efficiency of electromagnetic signal image search.

[0102] Step S210 , in the knowledge base, the reference electromagnetic signal image or reference text content corresponding to the target reference image and text semantic feature is determined as the search result of the data to be searched.

[0103] For details of step S210, see Figure 1 Step S105 in the illustrated embodiment is not described in detail here.

[0104] See also Figure 3 as well as Figure 2 , if the data to be searched is an electromagnetic signal image, the text content corresponding to the electromagnetic signal image is obtained through a multimodal large model (a column of the image-text conversion model), and a text feature extraction model is used to obtain a text feature vector (a column of text semantic features), which is also text feature extraction; the image feature vector (a column of image semantic features) is extracted through the IME-Net model (also called the image feature extraction model), which is also image feature extraction, and when the features are fused, a unit matrix of the same dimension is concatenated as a substitute for the text semantic feature tensor to obtain V image-text feature fusion (a column of image-text semantic features). First, based on the formula Determine the degree of membership between the semantic features of the image and text and the semantic features of the reference image and text corresponding to each central node, find the central node with the highest degree of membership, and then search the search graph in the search graph with the central node as the search starting point to obtain k results. According to the ID corresponding to the reference image and text in the result (a column of text identification information or electromagnetic signal image identification information), query the database (a column of the knowledge base) and return the final search result.

[0105] Supports converting electromagnetic high-level data structure text (a column of text data) into waveform-style electromagnetic signal images through a multimodal large model to enhance data utilization. After the waveform-style electromagnetic signal image is converted and generated, the matching image feature vector and text feature vector are obtained based on the image feature extraction model and the text feature extraction model for feature fusion to obtain V image-text feature fusion. Feature fusion, based on the NSG algorithm to build an image retrieval graph, and combined with the K-Means algorithm to build a rough search on the upper layer of the retrieval graph, to achieve efficient retrieval of high-level data text, and return the top N waveform graphs of vector similarity. Subsequently, the waveform style graph can be mined to further retrieve radar information, platform role, platform type, physique, working mode, collection location and other related data associated with similar waveform graphs.

[0106] In the field of electromagnetic space security, the data search method of electromagnetic signal images in the embodiment of the present application can provide intelligent support for system defense. The method obtains the corresponding text content by targeting the electromagnetic signal image of the signal, spectrum and / or interference source in the electromagnetic environment, and extracts the text semantic features of the text content, that is, the context features and the image semantic features of the electromagnetic signal image and performs feature fusion, and then performs a rough search followed by a detailed search to help deal with complex electromagnetic threats. The method can realize efficient processing of large-scale data. Specifically, a large number of signals, spectrum images and multidimensional data are involved in the electromagnetic space. Through semantic feature extraction and vector retrieval (that is, the search between the target image semantic features and the reference image semantic features), these complex data can be converted into compact vector representations, which can be quickly retrieved and matched on large-scale data sets. This not only significantly improves data processing efficiency, but also shortens analysis and response time, which is especially important in emergency situations.

[0107] This method can also achieve accurate matching of complex contexts: specifically, it extracts contextual information of electromagnetic signal images and text content through semantic feature extraction and vector retrieval. It not only focuses on single features in the image, but also can achieve more accurate matching and analysis by extracting multi-dimensional contextual information (such as spectral features, signal patterns, or propagation environment in text content). This is particularly important in electromagnetic threat identification and interference detection, and can effectively identify threats hidden in complex scenes.

[0108] This method can also support cross-modal data analysis: Specifically, through semantic feature extraction and vector retrieval, electromagnetic data from different sources (such as spectrum, communication signals or radar electromagnetic signal images, etc.) can be combined and processed and analyzed through a unified vector representation, eliminating the barriers between different data formats. This enables the system to more comprehensively analyze various information in the electromagnetic environment and provide richer intelligence support for threat identification and defense.

[0109] This method can also quickly discover similar patterns: in electromagnetic space, threat signals and interference patterns may recur in different forms. Through semantic feature extraction and vector retrieval, potential similar patterns or attack strategies can be discovered in a timely manner through rapid comparison with historical data (a column in the knowledge base), thereby achieving early identification and early warning of threats.

[0110] In an embodiment of the present application, when searching based on a retrieval graph, based on the degree of membership between the target graph semantic features and the reference graph semantic features corresponding to the central nodes of each group of nodes in the retrieval graph, the target central node is screened out as the retrieval starting point for searching. By determining the retrieval starting point, a rough search of the retrieval graph is first performed, and then a detailed search of the retrieval graph is performed from the retrieval starting point, thereby avoiding the traditional situation of searching the entire graph from a specific node, improving the retrieval efficiency of the retrieval graph, and further improving the efficiency of electromagnetic signal image search. And when searching, the semantic features of the target graph and text that combine the semantic features of the image and the semantic features of the text are considered, rather than the visual features of the electromagnetic signal image, to improve the accuracy of the electromagnetic signal image search.

[0111] Figure 7 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Figure 7 As shown, the electronic device 6 of this embodiment includes: at least one processor 60 ( Figure 7 Only one is shown in the figure) a processor, a memory 61, and a computer program 62 stored in the memory 61 and executable on the at least one processor 60, wherein the processor 60 implements the steps of any of the above-mentioned method embodiments when executing the computer program 62.

[0112] The electronic device 6 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The electronic device may include, but is not limited to, a processor 60 and a memory 61. Those skilled in the art will appreciate that Figure 7 It is only an example of the electronic device 6 and does not constitute a limitation on the electronic device 6. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0113] The processor 60 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0114] In some embodiments, the memory 61 may be an internal storage unit of the electronic device 6, such as a hard disk or memory of the electronic device 6. In other embodiments, the memory 61 may also be an external storage device of the electronic device 6, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 6. Further, the memory 61 may also include both an internal storage unit and an external storage device of the electronic device 6. The memory 61 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program. The memory 61 may also be used to temporarily store data that has been output or is to be output.

[0115] Corresponding to the data search method for electromagnetic signal graphics described in the above embodiment, Figure 8 A structural block diagram of a data search device for electromagnetic signal graphics provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0116] Reference Figure 8 , the device comprises: An acquisition module 100 is used to acquire a target electromagnetic signal image and target text content corresponding to the data to be searched using a picture-text generation model; An extraction module 200 is used to extract image semantic features of a target electromagnetic signal image and text semantic features of a target text content, wherein the image semantic features indicate electromagnetic signals in the target electromagnetic signal image, and the text semantic features indicate at least one of a spectrum feature, a signal mode, or a propagation environment of the electromagnetic signal; A fusion module 300 is used to fuse image semantic features and text semantic features to obtain target image and text semantic features; A search module 400 is used to search for a target reference image-text semantic feature corresponding to a target image-text semantic feature from a search graph corresponding to a knowledge base, wherein the knowledge base includes a plurality of reference electromagnetic signal images and a plurality of reference text contents, and the search graph includes a plurality of reference image-text semantic features, wherein one reference image-text semantic feature corresponds to one reference electromagnetic signal image or one reference text content; The determination module 500 is used to determine the reference electromagnetic signal image or reference text content corresponding to the target reference image and text semantic feature as the search result of the data to be searched in the knowledge base.

[0117] In some embodiments, the search graph is composed of multiple groups of nodes, each group of nodes includes a central node, and the search module 400 is used to: Determine the degree of membership between the reference image-text semantic features and the target image-text semantic features corresponding to each central node; Determine the central node corresponding to the membership degree that meets the first condition as the target node; Taking the target node as the search starting point, searching at least one node in the search graph until the stop condition is met, and obtaining the vector similarity between the reference image-text semantic feature corresponding to at least one node and the target image-text semantic feature; Based on the vector similarity and the reference image-text semantic feature corresponding to at least one node, a target reference image-text semantic feature is determined.

[0118] In some embodiments, the extraction module 200 is used to: Extract image semantic features from the target electromagnetic signal image using an image feature extraction model; The text feature extraction model is used to extract text semantic features from the target text content.

[0119] In some embodiments, the image feature extraction model includes a convolutional layer, a temporal layer, a spatial layer, and a fully connected layer; The convolution layer is used to extract local features of the target electromagnetic signal image and input the local features into the temporal layer and the spatial layer, where the local features indicate the electromagnetic signal; The time series layer is used to extract time series features from local features and input them into the fully connected layer. The time series features indicate the trend of electromagnetic signals changing over time. The spatial layer is used to extract spatial features from local features and input the spatial features into the fully connected layer. The spatial features include the trend of electromagnetic signals changing with space. The fully connected layer is used to perform weighted fusion of temporal features and spatial features to obtain image semantic features and output image semantic features.

[0120] In some embodiments, the fusion module 300 is used to: Perform feature alignment operations on the image semantic features and the text semantic features respectively to obtain new image semantic features and new text semantic features; Based on the text weight and image weight, the new image semantic features and the new text semantic features are weightedly fused to obtain the target image and text semantic features.

[0121] In some embodiments, the acquisition module 100 is used to: The image-text generation model is used to obtain the target electromagnetic signal image and target text content corresponding to the data to be searched, including: In the case where the data to be searched is an electromagnetic signal image, a target text content corresponding to the data to be searched is generated using a picture-text generation model, and the electromagnetic signal image is used as a target electromagnetic signal image; In the case where the data to be searched is text data, an electromagnetic signal image corresponding to the text data is generated using the image-text generation model, the generated electromagnetic signal image is used as the target electromagnetic signal image, and the text data is used as the target text content; When the data to be searched includes a first text material and a first electromagnetic signal image, a graphic-text generation model is used to generate a second electromagnetic signal image corresponding to the first text material, and to generate a first text content corresponding to the first electromagnetic signal image. The first electromagnetic signal image and the second electromagnetic signal image are used as target electromagnetic signal images, and the first text material and the first text content are used as target text content.

[0122] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0123] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0124] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0125] An embodiment of the present application provides a computer program product, which can implement the steps in the above-mentioned method embodiments when the computer program product is executed.

[0126] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the camera / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0127] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0128] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0129] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0130] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0131] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A data search method for electromagnetic signal graphics, characterized in that: include: The image-text generation model is used to obtain the target electromagnetic signal image and target text content corresponding to the data to be searched; Extracting image semantic features of the target electromagnetic signal image and text semantic features of the target text content, wherein the image semantic features indicate the electromagnetic signal in the target electromagnetic signal image, and the text semantic features indicate at least one of the spectrum features, signal mode or propagation environment of the electromagnetic signal; The image semantic features and the text semantic features are integrated to obtain the target image and text semantic features; Searching for a target reference image-text semantic feature corresponding to the target image-text semantic feature from a search graph corresponding to a knowledge base, wherein the knowledge base includes a plurality of reference electromagnetic signal images and a plurality of reference text contents, and the search graph includes a plurality of reference image-text semantic features, and one of the reference image-text semantic features corresponds to at least one of the reference electromagnetic signal images or at least one of the reference text contents; In the knowledge base, the reference electromagnetic signal image or reference text content corresponding to the target reference image-text semantic feature is determined as the search result of the data to be searched.

2. The method according to claim 1, characterized in that The search graph is composed of multiple groups of nodes, each group of nodes includes a central node, and the search graph corresponding to the knowledge base searches for the target reference image-text semantic feature corresponding to the target image-text semantic feature, including: Respectively determining the degree of membership between the reference image-text semantic feature corresponding to each of the central nodes and the target image-text semantic feature; Determine the central node corresponding to the membership degree that satisfies the first condition as the target node; Taking the target node as the search starting point, searching at least one node in the search graph until a stop condition is met, and obtaining a vector similarity between a reference image-text semantic feature corresponding to the at least one node and the target image-text semantic feature; The target reference image-text semantic feature is determined based on the vector similarity and the reference image-text semantic feature corresponding to the at least one node.

3. The method according to claim 2, characterized in that The step of extracting the image semantic features of the target electromagnetic signal image and the text semantic features of the target text content includes: Extracting the image semantic features from the target electromagnetic signal image using an image feature extraction model; The text semantic features are extracted from the target text content using a text feature extraction model.

4. The method according to claim 3, characterized in that The image feature extraction model includes a convolutional layer, a temporal layer, a spatial layer and a fully connected layer; The convolution layer is used to extract local features of the target electromagnetic signal image and input the local features to the temporal layer and the spatial layer, wherein the local features indicate the electromagnetic signal; The timing layer is used to extract timing features from the local features and input the timing features into the fully connected layer, wherein the timing features indicate a trend of the electromagnetic signal changing over time; The spatial layer is used to extract spatial features from the local features and input the spatial features into the fully connected layer, wherein the spatial features include the trend of the electromagnetic signal changing with space; The fully connected layer is used to perform weighted fusion of the temporal features and the spatial features to obtain the image semantic features, and output the image semantic features.

5. The method according to any one of claims 1 to 4, characterized in that: The fusing of the image semantic features and the text semantic features to obtain the image-text semantic features includes: Performing feature alignment operations on the image semantic features and the text semantic features respectively to obtain new image semantic features and new text semantic features; Based on the text weight and the image weight, the new image semantic feature and the new text semantic feature are weightedly fused to obtain the target image-text semantic feature.

6. The method according to claim 5, characterized in that The method of using the image-text generation model to obtain the target electromagnetic signal image and target text content corresponding to the data to be searched includes: In the case where the data to be searched is an electromagnetic signal image, the image-text generation model is used to generate target text content corresponding to the data to be searched, and the electromagnetic signal image is used as the target electromagnetic signal image; In the case where the data to be searched is text data, the electromagnetic signal image corresponding to the text data is generated by using the image-text generation model, the generated electromagnetic signal image is used as the target electromagnetic signal image, and the text data is used as the target text content; In the case where the data to be searched includes a first text material and a first electromagnetic signal image, the image-text generation model is used to generate a second electromagnetic signal image corresponding to the first text material, and to generate a first text content corresponding to the first electromagnetic signal image, the first electromagnetic signal image and the second electromagnetic signal image are used as the target electromagnetic signal images, the first text material is used as the target text content, and the first text content is used as the target text content.

7. A data search device for electromagnetic signal graphics, characterized in that: include: An acquisition module, used to acquire a target electromagnetic signal image and target text content corresponding to the data to be searched by using an image-text generation model; An extraction module, configured to extract image semantic features of the target electromagnetic signal image and text semantic features of the target text content, wherein the image semantic features indicate electromagnetic signals in the target electromagnetic signal image, and the text semantic features indicate at least one of spectrum features, signal modes or propagation environments of the electromagnetic signal; A fusion module, used for fusing the image semantic features and the text semantic features to obtain target image and text semantic features; A search module, used for searching a target reference image-text semantic feature corresponding to the target image-text semantic feature from a search graph corresponding to a knowledge base, wherein the knowledge base includes a plurality of reference electromagnetic signal images and a plurality of reference text contents, and the search graph includes a plurality of reference image-text semantic features, wherein one reference image-text semantic feature corresponds to one reference electromagnetic signal image or one reference text content; The determination module is used to determine, in the knowledge base, the reference electromagnetic signal image or the reference text content corresponding to the target reference image and text semantic feature as the search result of the data to be searched.

8. An electronic device, characterized in that: The electronic device comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the electronic device implements the data search method for electromagnetic signal graphics as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the data search method for electromagnetic signal graphs according to any one of claims 1 to 6 is implemented.

10. A computer program product, characterized in that The invention comprises a computer program, which, when being executed, enables the data search method of the electromagnetic signal pattern as claimed in any one of claims 1 to 6 to be executed.

Images