An image representation method and device based on multi-modal fusion and electronic equipment

Abstract

The application discloses a kind of based on multimodal fusion's image representation method, device and electronic equipment.Method includes: obtaining the image data of metal sample in electrolytic process, the total area of corrosion part is calculated, obtains first channel data;Electrochemical impedance spectroscopy of test metal is measured, the current density of corrosion metal is calculated, obtains second channel data;Obtain the sound spectrum diagram in the process of measuring out test metal electrochemical corrosion test, convert sound spectrum diagram into third channel data;Equivalent sampling is carried out, and RGB color space numerical value is obtained;Three channel data are fused into RGB single pixel point, then pixel point is stacked to form metal corrosion-resistant gradient map;Classification model is obtained by training, and the pre-classification of metal corrosion resistance is completed.The application can realize the pre-classification of experimental metal sample, reduce unnecessary repeated experimental labor, improve the efficiency of metal sample corrosion resistance experiment characterization, and can be widely applied in image processing technical field.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image representation method, apparatus and electronic device based on multimodal fusion. Background Technology

[0002] Electrochemical corrosion of metals is very common, significantly reducing their mechanical properties and damaging their structural shape. In seawater, metals primarily experience electrochemical corrosion. High-throughput preparation and characterization techniques are indispensable for developing metallic materials with high corrosion resistance.

[0003] In high-throughput experiments on seawater corrosion of metals, macroscopic high-throughput characterization techniques for metallic material corrosion are crucial for assessing the properties of metal samples, studying their corrosion characteristics, and understanding their corrosion patterns. Electrochemical corrosion techniques are commonly used for macroscopic high-throughput characterization of metallic material corrosion. Electrochemical corrosion experiments are used to study the corrosion characteristics of metal samples prepared through high-throughput methods, and to screen for metal samples with excellent corrosion resistance.

[0004] In related technologies, electrochemical corrosion experiments on metals typically employ multi-channel electrochemical workstations combined with optical measurement techniques to characterize the metal's properties. Multi-channel electrochemical workstations can simultaneously test the electrochemical behavior of multiple metal samples in different corrosive environments, such as measuring corrosion potential, corrosion current density, and impedance spectroscopy to characterize the metal's corrosion resistance. Furthermore, the corrosion process produces clearly visible corrosion areas on the image, which can be used to characterize the metal's corrosion resistance. In addition, severe corrosion is accompanied by high-amplitude acoustic spectra such as bubbles and flaking, which can be collected and analyzed to characterize the metal's corrosion resistance. Currently, in metal electrochemical corrosion experiments, professional technicians primarily analyze electrochemical corrosion spectra to screen for metal samples with excellent corrosion resistance. However, traditional metal corrosion experiments suffer from limitations such as high repeatability, large data acquisition volumes, and a low proportion of high-quality metal samples. Summary of the Invention

[0005] The main objective of this invention is to propose an image characterization method, device, and electronic device based on multimodal fusion, so as to achieve pre-classification of experimental metal samples, reduce unnecessary repetitive experimental work, and improve the efficiency of corrosion resistance experimental characterization of metal samples.

[0006] To achieve the above objectives, one aspect of this invention proposes an image representation method based on multimodal fusion, comprising the following steps:

[0007] Image data of metal samples during electrolysis is acquired, and the total area of ​​the corroded part is calculated by edge detection and shape recognition. The obtained total area is then processed into the first channel data.

[0008] The electrochemical impedance spectroscopy of the test metal was measured, and the current density of the corroded metal was calculated by combining the corrosion current density formula with the electrochemical impedance spectroscopy. The obtained current density was then processed into second channel data.

[0009] Acquire the acoustic spectrum of the test metal during the electrochemical corrosion test, and convert the acoustic spectrum into third channel data using weight allocation;

[0010] The first channel data, the second channel data, and the third channel data are sampled equally to obtain RGB color space values;

[0011] The first channel data, the second channel data, and the third channel data are fused into a single RGB pixel, and then the pixels are stacked to form a metal corrosion resistance gradient image.

[0012] A classification model is trained based on the metal corrosion resistance gradient map, and the metal corrosion resistance performance is pre-classified based on the classification model.

[0013] In some embodiments, acquiring image data of a metal sample during electrolysis, calculating the total area of ​​the corroded portion through edge detection and shape recognition, and processing the obtained total area into first channel data includes the following steps:

[0014] Two vision cameras were used to acquire electrochemical corrosion images of the metal block sample at two diagonal angles.

[0015] For each acquired electrochemical corrosion image, the Threshold binarization algorithm was used for binarization processing;

[0016] Use the findContours() and drawContours() functions in OpenCV for edge detection and contour processing;

[0017] The cv.contourArea() function is used to solve for the contour area, and the total area of ​​the eroded part is calculated.

[0018] The total area of ​​the corroded portion is processed into first-channel data of corrosion area-time.

[0019] In some embodiments, the electrochemical impedance spectroscopy of the test metal is measured, and the current density of the corroded metal is calculated using the corrosion current density formula combined with the electrochemical impedance spectroscopy. The obtained current density is then processed into second-channel data, including the following steps:

[0020] Electrochemical impedance spectroscopy (EIS) of the corroded metal was acquired using a multi-channel electrochemical workstation. The EIS data were then fitted using a Randle circuit to obtain the electrolyte resistance R. s Double-layer capacitance C dl and polarization resistance R p ;

[0021] The calculated Stern-Geary constant B;

[0022] According to the polarization resistance R obtained from the fitting p And based on the Stern-Geary constant B, the corrosion current density j is calculated. corr ;

[0023] The formula for calculating the Stern-Geary constant B is as follows:

[0024]

[0025] Corrosion current density j corr The calculation formula is:

[0026]

[0027] Where, β a It is the Tafel slope of the anode, β c It is the cathode Tafel slope; j corr It is the corrosion current density; R p B is the polarization resistance; B is the Stern-Geary constant.

[0028] In some embodiments, acquiring the acoustic spectrum during the electrochemical corrosion test of the test metal and converting the acoustic spectrum into third-channel data using weight allocation includes the following steps:

[0029] The acoustic spectrum data of the test metal during the electrochemical corrosion test was received and measured by an acoustic spectrum analyzer; the acoustic spectrum data contains two data points: frequency and amplitude.

[0030] The acoustic spectrum data were linearly weighted and superimposed using a weighting of 0.05 for frequency and 0.95 for amplitude. The resulting data was labeled as corrosion intensity and was used as the third channel data.

[0031] In some embodiments, the step of sampling the first channel data, the second channel data, and the third channel data equally to obtain RGB color space values ​​includes the following steps:

[0032] For each group of metal samples, 25,000,000 data samples were collected during the corrosion test.

[0033] The first channel data, the second channel data, and the third channel data are processed to obtain RGB color space values. The expression for this processing is as follows:

[0034]

[0035] Where x1(t) represents the first channel data, the second channel data, and the third channel data after numerical processing in RGB color space, x(t) represents the original first channel data, the second channel data, and the third channel data, and the subscripts min and max represent the minimum and maximum values ​​of the current dataset, respectively.

[0036] In some embodiments, fusing the first channel data, the second channel data, and the third channel data into a single RGB pixel, and then stacking the pixels to form a metal corrosion-resistant gradient image, includes the following steps:

[0037] After aligning the three-channel data, which have been numerically processed in the color space, according to the time scale, the numerical values ​​of the color space are then fused.

[0038] The data set of 25,000,000*3 is merged into 25,000,000 color pixels;

[0039] The 25,000,000 pixels are stacked in a 5,000*5,000 pixel format to form a metal corrosion-resistant gradient image.

[0040] In some embodiments, the step of training a classification model based on the metal corrosion resistance gradient map and pre-classifying the metal corrosion resistance performance based on the classification model includes the following steps:

[0041] Based on the actual quality of the metal samples, a predetermined number of metal samples are calibrated and divided into three grades.

[0042] Corrosion resistance gradient diagrams of each sample are correlated with metal corrosion resistance calibration.

[0043] Transfer learning is performed using a pre-trained ResNet image classification model, replacing the last layer of the pre-trained model with a model suitable for classifying gradient images of metal corrosion resistance using the train_model() function module;

[0044] Define the Incremental_Learning() function module, which is used to perform incremental learning, so that the calibration data in subsequent metal experiments can be added to the model for the model to learn, and to continuously optimize the classification model.

[0045] The corrosion resistance of metals is pre-classified based on the classification model.

[0046] Another aspect of the present invention provides an image representation device based on multimodal fusion, comprising:

[0047] The first module is used to acquire image data of metal samples during electrolysis, calculate the total area of ​​the corroded part through edge detection and shape recognition, and process the obtained total area into the first channel data.

[0048] The second module is used to measure the electrochemical impedance spectroscopy of the test metal. It calculates the current density of the corroded metal using the corrosion current density formula combined with the electrochemical impedance spectroscopy, and processes the obtained current density into second channel data.

[0049] The third module is used to acquire the acoustic spectrum of the measured metal during the electrochemical corrosion test, and convert the acoustic spectrum into third channel data using weight allocation;

[0050] The fourth module is used to sample the first channel data, the second channel data, and the third channel data equally to obtain RGB color space values;

[0051] The fifth module is used to fuse the first channel data, the second channel data and the third channel data into RGB single pixels, and then stack the pixels to form a metal corrosion resistance gradient image;

[0052] The sixth module is used to train a classification model based on the metal corrosion resistance gradient map, and to pre-classify the metal corrosion resistance performance based on the classification model.

[0053] Another aspect of the present invention provides an electronic device, including a processor and a memory;

[0054] The memory is used to store programs;

[0055] The processor executes the program to implement the method described above.

[0056] Another aspect of this invention provides a computer-readable storage medium storing a program that is executed by a processor to implement the methods described above.

[0057] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned method.

[0058] The embodiments of this invention include at least the following beneficial effects: This invention provides an image characterization method, device, and electronic device based on multimodal fusion. This invention acquires image data of metal samples during electrolysis, calculates the total area of ​​the corroded portion through edge detection and shape recognition, and processes the obtained total area into first-channel data; measures the electrochemical impedance spectroscopy of the test metal, calculates the current density of the corroded metal using the corrosion current density formula combined with the electrochemical impedance spectroscopy, and processes the obtained current density into second-channel data; acquires and measures the acoustic spectrum of the test metal during the electrochemical corrosion test, and converts the acoustic spectrum into third-channel data using weight allocation; equally samples the first-channel data, second-channel data, and third-channel data to obtain RGB color space values; fuses the first-channel data, second-channel data, and third-channel data into RGB single pixels, and then stacks the pixels to form a metal corrosion resistance gradient map; trains a classification model based on the metal corrosion resistance gradient map, and pre-classifies the metal corrosion resistance performance based on the classification model. This invention can achieve pre-classification of experimental metal samples, reduce unnecessary repetitive experimental work, and improve the efficiency of metal sample corrosion resistance experimental characterization. Attached Figure Description

[0059] Figure 1 This is a schematic diagram of an implementation environment provided by an embodiment of the present invention;

[0060] Figure 2 This is a flowchart of the overall steps provided in the embodiments of the present invention;

[0061] Figure 3 This is a schematic diagram of pixel stacking provided in an embodiment of the present invention;

[0062] Figure 4 This is a schematic diagram illustrating the implementation of the Incremental_Learning() function module provided in this embodiment of the invention;

[0063] Figure 5 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of this invention; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this invention as detailed in the appended claims.

[0065] It is understood that the terms “first,” “second,” etc., used in this invention may be used herein to describe various concepts, but unless specifically stated otherwise, these concepts are not limited by these terms. These terms are used only to distinguish one concept from another. For example, first information may also be referred to as second information without departing from the scope of embodiments of the invention, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to determination” as used herein may be interpreted as “when…” or “when…” or “in response to determination.”

[0066] The terms “at least one,” “multiple,” “each,” “any,” etc., used in this invention, “at least one” includes one, two, or more than two; “multiple” includes two or more than two; “each” refers to each of the corresponding multiple; and “any” refers to any one of the multiple.

[0067] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to limit the invention.

[0068] The image representation method, apparatus, and electronic device based on multimodal fusion provided in this invention relates to the field of image processing technology. The image representation method based on multimodal fusion provided in this invention can be applied to a terminal, a server, or software running on a terminal or server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or in-vehicle terminal, but is not limited thereto; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application implementing the image representation method based on multimodal fusion, but is not limited to the above forms.

[0069] This invention can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0070] like Figure 1 The diagram shown is a schematic representation of an implementation environment provided by an embodiment of the present invention. (Refer to...) Figure 1 The implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected via a network, either wirelessly or via a wired connection, to complete data transmission and exchange.

[0071] Server 101 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0072] Additionally, server 101 can also be a node server in a blockchain network. Blockchain is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms.

[0073] Terminal 102 can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc. It can also be a vehicle-mounted terminal of the various device types described above, but is not limited to these. Terminal 102 and server 101 can be directly or indirectly connected via wired or wireless communication, and this embodiment of the invention does not impose any limitations.

[0074] Exemplary based on Figure 1 The implementation environment shown in this embodiment of the invention provides an image representation method based on multimodal fusion. The following description uses the application of this image representation method based on multimodal fusion in server 101 as an example. It can be understood that this method can also be applied to terminal 102.

[0075] Reference Figure 2 , Figure 2 The flowchart illustrates a multimodal fusion-based image representation method applied to a server, as provided in this embodiment of the invention. The execution subject of this method can be any of the aforementioned computer devices (including a server or a terminal). (Refer to...) Figure 2 The method may include the following steps:

[0076] Image data of metal samples during electrolysis is acquired, and the total area of ​​the corroded part is calculated by edge detection and shape recognition. The obtained total area is then processed into the first channel data.

[0077] The electrochemical impedance spectroscopy of the test metal was measured, and the current density of the corroded metal was calculated by combining the corrosion current density formula with the electrochemical impedance spectroscopy. The obtained current density was then processed into second channel data.

[0078] Acquire the acoustic spectrum of the test metal during the electrochemical corrosion test, and convert the acoustic spectrum into third channel data using weight allocation;

[0079] The first channel data, the second channel data, and the third channel data are sampled equally to obtain RGB color space values;

[0080] The first channel data, the second channel data, and the third channel data are fused into a single RGB pixel, and then the pixels are stacked to form a metal corrosion resistance gradient image.

[0081] A classification model is trained based on the metal corrosion resistance gradient map, and the metal corrosion resistance performance is pre-classified based on the classification model.

[0082] Specifically, this invention addresses the problem of low efficiency in metal electrochemical corrosion experiments by proposing a multimodal fusion three-channel composite image characterization algorithm that integrates visual, electrochemical, and auditory features. This algorithm pre-classifies the metal samples used in the experiment, thereby improving the efficiency of the corrosion test. (The above is an appendix.) Figure 2 The overall implementation steps are shown in the document, and the specific implementation process of each step includes:

[0083] Step 1: Image data of the metal sample during electrolysis is acquired using two vision cameras. The total area of ​​the corroded portion is calculated through edge detection and shape recognition. Finally, the obtained area data is processed into first-channel data, such as corrosion area minus time.

[0084] Step 2: The electrochemical impedance spectroscopy (EIS) of the test metal was measured using a multi-channel electrochemical workstation. The current density of the corroded metal was calculated using the corrosion current density formula combined with the EIS. Finally, the obtained current density data was processed into a second channel data format: current density - time.

[0085] Step 3: Obtain the acoustic spectrum data of the test metal during the electrochemical corrosion test using an acoustic spectrum analyzer. Then, convert the acoustic spectrum into third-channel data (corrosion intensity-time) using weighted allocation.

[0086] Step 4: Sample the data from the three channels 25,000,000 times in equal increments, and then process the sampled data into RGB color space values ​​after uniform normalization.

[0087] Step 5: Merge the 25000000*3 structure data of the three channels into RGB individual pixels, and then stack the pixels to form a metal anti-corrosion gradient image.

[0088] Step 6: Classify the metal corrosion resistance into three levels. Based on the pre-trained classification model corresponding to the metal corrosion resistance gradient map obtained from pre-measured and calibrated corrosion resistance data, pre-classify the metal corrosion resistance into three levels.

[0089] In some embodiments, step 1 uses two vision cameras to acquire electrochemical corrosion images of the metal block sample at two diagonal angles. Each acquired image is then binarized using a thresholding algorithm. Edge detection and contour processing are then performed using the `findContours()` and `drawContours()` functions in OpenCV. Finally, the `cv.contourArea()` function is used to calculate the contour area.

[0090] Since only two vision cameras are used to detect the corrosion area of ​​the metal sample, and the angle between the camera and the plane of the bulk metal is 45°, the actual corrosion area S(t) calculated using the above algorithm is 2*S(t). S(t) represents the change of the corrosion area with the electrochemical test time and is the data from the first channel.

[0091] In some embodiments, step 2 uses a multi-channel electrochemical workstation to acquire the electrochemical impedance spectroscopy (EIS) of the corroded metal, and uses Randle's circuit to fit the EIS data to obtain the electrolyte resistance R. s Double-layer capacitance C dl and polarization resistance R p The polarization resistance R obtained from the fitting is... p The Stern-Geary constant B, obtained by formula (1.1), is used to calculate the corrosion current density j using formula (1.2). corr .

[0092]

[0093] Where, β a It is the Tafel slope of the anode, β c It is the cathode Tafel slope.

[0094]

[0095] Where, j corr It is the corrosion current density, R p It is the polarization resistance, and B is the Stern-Geary constant.

[0096] The calculated second channel data is then j. corr (t).

[0097] In some embodiments, step 3 involves receiving and measuring the acoustic spectrum data during the electrochemical corrosion test of the test metal using an acoustic spectrum analyzer. The acoustic spectrum data contains two data points: frequency and amplitude. We use a weighting of 0.05 for frequency and 0.95 for amplitude to perform linear weighted superposition processing on the acoustic spectrum data. The processed data is labeled as corrosion intensity Q(t), and is the third channel data.

[0098] In some embodiments, step 4 involves sampling data 25,000,000 times for each group of metal samples during corrosion testing. After processing the data from the three channels through steps 1, 2, and 3, the dataset S(t),j is obtained. corr Q(t) and Q(t). Then, we use formula (1.3) to perform RGB color space numerical processing on the three datasets above.

[0099]

[0100] Where x1(t) represents S(t) after numerical processing in the RGB color space, j corr The datasets S(t) and Q(t) are given, where x(t) represents the original dataset S(t), and j is the number of elements in the dataset. corr The subscripts min and max represent the minimum and maximum values ​​of the current dataset, respectively.

[0101] In some embodiments, step 5 aligns the three-channel data after color space numerical processing in step 4 according to the time scale, and then performs color space numerical fusion. The 25,000,000*3 data set is fused into 25,000,000 color pixels. Then, these 25,000,000 pixels are stacked in a 5000*5000 pixel array. The stacking method is as follows: Figure 3 As shown.

[0102] In some embodiments, step 6 first involves manually calibrating a certain number of metal samples based on their actual quality, dividing them into three grades, and then mapping the metal corrosion resistance gradient map generated in step 5 to the metal corrosion resistance calibration. First, we use a pre-trained ResNet image classification model for transfer learning, replacing the last layer of the pre-trained model with a model suitable for classifying the metal corrosion resistance gradient map using the `train_model()` function module. Then, we define the `Incremental_Learning()` function module to implement subsequent incremental learning, meaning that calibration data from subsequent metal experiments can be added to the model for continuous learning and optimization of the classification model.

[0103] Regarding the definition of Incremental_Learning(), as follows: Figure 4 As shown, the Incremental_Learning() function module in this embodiment of the invention can be used to obtain new training data and validation sets, define loss functions and optimizers, fine-tune the model, and validate the model.

[0104] Suppose we have initial datasets in 'path_to_init_ial_train_data' and 'path_to_init_ial_val_data', and new datasets in 'path_to_new_train_data' and 'path_to_val_data'. We can add the new 'path_to_new_train_data' and 'path_to_val_data' datasets to a pre-trained model to optimize the previous model, thus achieving incremental learning.

[0105] In summary, this invention is rationally designed. The multimodal fusion three-channel composite image characterization algorithm model, which integrates visual, electrochemical, and auditory features, can effectively improve the characterization efficiency of metal corrosion resistance tests during high-throughput preparation and characterization of metallic materials. (Through pre-classification, professionals can further subdivide high-quality categories when analyzing metal corrosion resistance, reducing labor costs.) This reduces the input of labor costs and improves the level of automation in characterization. Furthermore, because multimodal classification is fused, the classification effect is significant, reducing the likelihood of overlooking superior material samples due to one-sided classification features, which is common in single-modal classification algorithms.

[0106] This invention also provides an image representation device based on multimodal fusion, comprising:

[0107] The first module is used to acquire image data of metal samples during electrolysis, calculate the total area of ​​the corroded part through edge detection and shape recognition, and process the obtained total area into the first channel data.

[0108] The second module is used to measure the electrochemical impedance spectroscopy of the test metal. It calculates the current density of the corroded metal using the corrosion current density formula combined with the electrochemical impedance spectroscopy, and processes the obtained current density into second channel data.

[0109] The third module is used to acquire the acoustic spectrum of the measured metal during the electrochemical corrosion test, and convert the acoustic spectrum into third channel data using weight allocation;

[0110] The fourth module is used to sample the first channel data, the second channel data, and the third channel data equally to obtain RGB color space values;

[0111] The fifth module is used to fuse the first channel data, the second channel data and the third channel data into RGB single pixels, and then stack the pixels to form a metal corrosion resistance gradient image;

[0112] The sixth module is used to train a classification model based on the metal corrosion resistance gradient map, and to pre-classify the metal corrosion resistance performance based on the classification model.

[0113] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0114] This invention also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned image representation method based on multimodal fusion. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0115] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0116] Please see Figure 5 , Figure 5 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0117] The processor 501 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.

[0118] The memory 502 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 502 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 502 and is called and executed by the processor 501 to execute the image representation method based on multimodal fusion of the embodiments of this invention.

[0119] The input / output interface 503 is used to implement information input and output;

[0120] The communication interface 504 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0121] Bus 505 transmits information between various components of the device (e.g., processor 501, memory 502, input / output interface 503, and communication interface 504);

[0122] The processor 501, memory 502, input / output interface 503, and communication interface 504 are connected to each other within the device via bus 505.

[0123] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described image representation method based on multimodal fusion.

[0124] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0125] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0126] The embodiments described in this invention are for the purpose of more clearly illustrating the technical solutions of the embodiments of this invention, and do not constitute a limitation on the technical solutions provided by the embodiments of this invention. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this invention are also applicable to similar technical problems.

[0127] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present invention, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0128] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0129] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0130] The terms "first," "second," "third," "fourth," etc. (if present) in the specification and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0131] It should be understood that in this invention, "at least one (item)" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0132] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0133] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0134] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0135] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0136] The preferred embodiments of the present invention have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and spirit of the present invention should be within the scope of the claims of the present invention.

Claims

1. An image representation method based on multimodal fusion, characterized in that, Includes the following steps: Image data of metal samples during electrolysis is acquired, and the total area of ​​the corroded part is calculated by edge detection and shape recognition. The obtained total area is then processed into the first channel data. The electrochemical impedance spectroscopy of the test metal was measured, and the current density of the corroded metal was calculated by combining the corrosion current density formula with the electrochemical impedance spectroscopy. The obtained current density was then processed into second channel data. Acquire the acoustic spectrum of the test metal during the electrochemical corrosion test, and convert the acoustic spectrum into third channel data using weight allocation; The first channel data, the second channel data, and the third channel data are sampled equally to obtain RGB color space values; The first channel data, the second channel data, and the third channel data are fused into a single RGB pixel, and then the pixels are stacked to form a metal corrosion resistance gradient image. A classification model is trained based on the metal corrosion resistance gradient map, and the metal corrosion resistance performance is pre-classified based on the classification model. The electrochemical impedance spectroscopy of the test metal is measured, and the current density of the corroded metal is calculated using the corrosion current density formula combined with the electrochemical impedance spectroscopy. The obtained current density is then processed into second-channel data, including the following steps: Electrochemical impedance spectroscopy (EIS) of the corroded metal was acquired using a multichannel electrochemical workstation. Randle's circuit was used to fit the EIS data to obtain the electrolyte resistance R. s Double-layer capacitance C dl and polarization resistance R p ; The calculated Stern-Geary constant B; According to the polarization resistance R obtained from the fitting p And based on the Stern-Geary constant B, the corrosion current density j is calculated. corr ; The formula for calculating the Stern-Geary constant B is as follows: ， Corrosion current density j corr The calculation formula is: ， in, It is the Tafel slope of the anode. It is the cathode Tafel slope; j corr It is the corrosion current density; R p It is the polarization resistance; B is the Stern-Geary constant; The process of acquiring and measuring the acoustic spectrum during the electrochemical corrosion test of the test metal, and converting the acoustic spectrum into third-channel data using weight allocation, includes the following steps: The acoustic spectrum data of the test metal during the electrochemical corrosion test was received and measured by an acoustic spectrum analyzer; the acoustic spectrum data contains two data points: frequency and amplitude. The acoustic spectrum data were linearly weighted and superimposed using a weighting of 0.05 for frequency and 0.95 for amplitude. The resulting data was labeled as corrosion intensity and was used as the third channel data. The process of fusing the first channel data, the second channel data, and the third channel data into a single RGB pixel, and then stacking these pixels to form a metal corrosion-resistant gradient image, includes the following steps: After aligning the three-channel data, which have been numerically processed in the color space, according to the time scale, the numerical values ​​of the color space are then fused. 25,000,000 The data set of 3 was merged into 25,000,000 color pixels; Divide the 25,000,000 pixels into 5,000... 5000 pixels are stacked to form a metal corrosion-resistant gradient image.

2. The image representation method based on multimodal fusion according to claim 1, characterized in that, The process of acquiring image data of the metal sample during electrolysis, calculating the total area of ​​the corroded portion through edge detection and shape recognition, and processing the obtained total area into first-channel data includes the following steps: Two vision cameras were used to acquire electrochemical corrosion images of the metal block sample at two diagonal angles. For each acquired electrochemical corrosion image, a threshold binarization algorithm was used for binarization. Use the findContours() and drawContours() functions in OpenCV for edge detection and contour processing; The cv.contourArea() function is used to solve for the contour area, and the total area of ​​the eroded part is calculated. The total area of ​​the corroded portion is processed into first-channel data of corrosion area-time.

3. The image representation method based on multimodal fusion according to claim 1, characterized in that, The step of sampling the first channel data, the second channel data, and the third channel data equally to obtain RGB color space values ​​includes the following steps: For each group of metal samples, 25,000,000 data samples were collected during the corrosion test. The first channel data, the second channel data, and the third channel data are processed to obtain RGB color space values. The expression for this processing is as follows: ， in, This represents the first channel data, the second channel data, and the third channel data after numerical processing in the RGB color space. This represents the original data from the first channel, the second channel, and the third channel. The subscripts min and max represent the minimum and maximum values ​​of the current dataset, respectively.

4. The image representation method based on multimodal fusion according to claim 1, characterized in that, The process of training a classification model based on the metal corrosion resistance gradient map and pre-classifying the metal corrosion resistance performance based on the classification model includes the following steps: Based on the actual quality of the metal samples, a predetermined number of metal samples are calibrated and divided into three grades. Corrosion resistance gradient diagrams of each sample are correlated with metal corrosion resistance calibration. Transfer learning is performed using a pre-trained ResNet image classification model, replacing the last layer of the pre-trained model with a model suitable for classifying gradient images of metal corrosion resistance using the train_model() function module; Define the Incremental_Learning() function module, which is used to perform incremental learning to continue adding calibration data from subsequent metal experiments to the model for learning, thereby completing the continuous optimization of the classification model; The corrosion resistance of metals is pre-classified based on the classification model.

5. An apparatus for implementing the image representation method based on multimodal fusion as described in any one of claims 1-4, characterized in that, include: The first module is used to acquire image data of metal samples during electrolysis, calculate the total area of ​​the corroded part through edge detection and shape recognition, and process the obtained total area into the first channel data. The second module is used to measure the electrochemical impedance spectroscopy of the test metal. It calculates the current density of the corroded metal using the corrosion current density formula combined with the electrochemical impedance spectroscopy, and processes the obtained current density into second channel data. The third module is used to acquire the acoustic spectrum of the measured metal during the electrochemical corrosion test, and convert the acoustic spectrum into third channel data using weight allocation; The fourth module is used to sample the first channel data, the second channel data, and the third channel data equally to obtain RGB color space values; The fifth module is used to fuse the first channel data, the second channel data and the third channel data into RGB single pixels, and then stack the pixels to form a metal corrosion resistance gradient image; The sixth module is used to train a classification model based on the metal corrosion resistance gradient map, and to pre-classify the metal corrosion resistance performance based on the classification model.

6. An electronic device, characterized in that, Including the processor and memory; The memory is used to store programs; The processor executes the program to implement the method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The storage medium stores a program that is executed by a processor to implement the method as described in any one of claims 1 to 4.

Images