Image representation method and device based on multi-modal fusion and electronic equipment

Through the multimodal fusion image characterization method, combined with image data, electrochemical impedance spectrum and sound spectrum, the pre-classification of metal samples is solved, and the inefficiency problem in traditional methods is improved and the efficiency of metal corrosion resistance experiments is improved.

CN118941904AActive Publication Date: 2024-11-12SOUTHERN MARINE SCI & ENG GUANGDONG LAB (ZHUHAI) +1

Patent Information

Application Number
CN202410981141.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-22
Publication Date
2024-11-12
Estimated Expiration
2044-07-22

AI Technical Summary

Technical Problem

In metal electrochemical corrosion experiments, traditional methods have problems such as high repeatability, large data collection and low proportion of high-quality metal samples, resulting in inefficiency.

Method used

Using a multimodal fusion-based image characterization method, the image data, electrochemical impedance spectrum and sound spectrum diagram during the electrolysis process are fused into RGB color space numerical values ​​in combination with edge detection, shape recognition, current density calculation and weight allocation. , form a metal corrosion resistance gradient map and pre-classify it through classification model.

Benefits of technology

Pre-classification of metal samples is achieved, unnecessary repeated experimental labor is reduced, and the efficiency of metal samples to resist corrosion experimental characterization is improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118941904A_ABST
    Figure CN118941904A_ABST
Patent Text Reader

Abstract

The invention discloses an image characterization method and device based on multi-modal fusion and electronic equipment. The method comprises the following steps: acquiring image data of a metal sample in an electrolysis process, and calculating the total area of a corrosion part to obtain first channel data; measuring the electrochemical impedance spectrum of the test metal, and calculating the current density of the corrosion metal to obtain second channel data; acquiring and measuring a spectrogram in the electrochemical corrosion test process of the test metal, and converting the spectrogram into third channel data; carrying out equal-amount sampling to obtain an RGB color space numerical value; the data of the three channels are fused into RGB single pixel points, and then pixel points are stacked to form a metal anti-corrosion gradient graph; a classification model is obtained through training, and pre-classification of the corrosion resistance of the metal is completed. According to the method, the experimental metal samples can be pre-classified, unnecessary repeated experimental labor is reduced, the efficiency of metal sample corrosion resistance experimental characterization is improved, and the method can be widely applied to the technical field of image processing.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image representation method, device and electronic equipment based on multimodal fusion. Background Art

[0002] Electrochemical corrosion of metals is a very common phenomenon. It can significantly reduce the mechanical properties of metals and destroy their structural shape. The main form of corrosion encountered by metals in seawater is electrochemical corrosion. In order to develop metal materials with high corrosion resistance, high-throughput preparation and characterization technology is indispensable.

[0003] In the high-throughput experiment of seawater corrosion of metals, the high-throughput characterization technology of macroscopic metal material corrosion is an important means to detect the performance of metal samples, study the corrosion characteristics of metal samples, and master their corrosion laws. Electrochemical corrosion technology is commonly used in high-throughput characterization of macroscopic metal material corrosion. The corrosion characteristics of high-throughput prepared metal samples are studied through electrochemical corrosion experiments. Metal samples with excellent corrosion resistance are screened out.

[0004] In related technologies, electrochemical corrosion experiments of metals usually use multi-channel electrochemical workstations combined with optical measurement and other technologies to characterize the properties of metals. Multi-channel electrochemical workstations can simultaneously test the electrochemical behavior of multiple metal samples in different corrosion environments, such as measuring corrosion potential, corrosion current density and impedance spectrum to characterize the corrosion resistance of metals. On the other hand, during the metal corrosion process, obvious corrosion areas will appear on the image, and the corrosion resistance of the metal can be characterized by the corrosion image. In addition, the severe corrosion process will also be accompanied by the generation of high-jump sound spectra such as bubbles and shedding, and the corrosion resistance of the metal can be characterized by sound spectrum collection and analysis. At present, in metal electrochemical corrosion experiments, professional technicians are mainly used to analyze the electrochemical corrosion spectrum to screen metal samples with excellent corrosion resistance. However, traditional metal corrosion experiments have the problems of high repeatability, huge data collection volume and low proportion of high-quality metal samples. Summary of the invention

[0005] The main purpose of the embodiments of the present invention is to propose an image characterization method, device and electronic equipment based on multimodal fusion to achieve pre-classification of experimental metal samples, reduce unnecessary repetitive experimental labor, and improve the efficiency of corrosion resistance experimental characterization of metal samples.

[0006] To achieve the above object, an embodiment of the present invention provides an image representation method based on multimodal fusion, comprising the following steps:

[0007] Acquire image data of the metal sample during the electrolysis process, calculate the total area of ​​the corroded part through edge detection and shape recognition, and process the obtained total area into first channel data;

[0008] Measure the electrochemical impedance spectrum of the test metal, calculate the current density of the corroded metal using the corrosion current density formula combined with the electrochemical impedance spectrum, and process the obtained current density into the second channel data;

[0009] Obtaining and measuring the acoustic spectrogram during the electrochemical corrosion test of the test metal, and converting the acoustic spectrogram into third channel data by weight allocation;

[0010] The first channel data, the second channel data and the third channel data are equally sampled to obtain RGB color space values;

[0011] The first channel data, the second channel data and the third channel data are merged into a single RGB pixel point, and then the pixel points are stacked to form a metal anti-corrosion gradient map;

[0012] A classification model is obtained by training the metal corrosion resistance gradient map, and the metal corrosion resistance performance is pre-classified according to the classification model.

[0013] In some embodiments, the method of acquiring image data of a metal sample during electrolysis, calculating the total area of ​​the corroded portion by edge detection and shape recognition, and processing the obtained total area into first channel data comprises the following steps:

[0014] Using two visual cameras, electrochemical corrosion images of metal block samples at two diagonal angles are obtained;

[0015] Each electrochemical corrosion image obtained is binarized using the threshold binarization algorithm;

[0016] Use findContours() and drawContours() functions in opencv for edge detection and contour processing;

[0017] Use cv2.contourArea() function to solve the contour area and calculate the total area of ​​the corroded part;

[0018] The first channel data is processed into corrosion area-time according to the total area of ​​the corroded part.

[0019] In some embodiments, the electrochemical impedance spectrum of the test metal is measured, the current density of the corroded metal is calculated using the corrosion current density formula combined with the electrochemical impedance spectrum, and the obtained current density is processed into second channel data, including the following steps:

[0020] The electrochemical impedance spectrum of the corroded metal was obtained using a multi-channel electrochemical workstation, and the Rand le's circuit was used to fit the electrochemical impedance spectrum data to obtain the electrolyte resistance R s , double layer capacitance C dl and polarization resistance R p ;

[0021] The calculated Stern-Geary constant B;

[0022] According to the fitting polarization resistance R p And according to the Stern-Geary constant B, the corrosion current density j is calculated corr ;

[0023] Among them, the calculation formula of Stern-Geary constant B is:

[0024]

[0025] Corrosion current density j corr The calculation formula is:

[0026]

[0027] Among them, β a is the anodic Tafel slope, β c is the cathode Tafel slope; j corr is the corrosion current density; R p is the polarization resistance; B is the Stern-Geary constant.

[0028] In some embodiments, the step of obtaining and measuring a spectrogram during the electrochemical corrosion test of the test metal, and converting the spectrogram into third channel data by weight allocation, comprises the following steps:

[0029] The spectrum data of the test metal electrochemical corrosion test is received and measured by the spectrum analyzer; wherein the spectrum data includes two data: frequency and amplitude;

[0030] The acoustic spectrum data is processed by linear weighted superposition using a weight distribution of frequency 0.05 and amplitude 0.95. The data obtained after processing is calibrated as corrosion intensity and is the third channel data.

[0031] In some embodiments, the step of equally sampling the first channel data, the second channel data, and the third channel data to obtain RGB color space values ​​comprises the following steps:

[0032] 25,000,000 data samples were collected for each group of metal sample corrosion tests;

[0033] The first channel data, the second channel data and the third channel data are numerically processed in RGB color space to obtain RGB color space values. The expression of the processing process is:

[0034]

[0035] Among them, x1(t) represents the first channel data, the second channel data and the third channel data after numerical processing in RGB color space, x(t) represents the original first channel data, the second channel data and the third channel data, and the subscripts min and max represent the minimum and maximum values ​​of the current data set, respectively.

[0036] In some embodiments, the step of fusing the first channel data, the second channel data, and the third channel data into a single RGB pixel, and then stacking the pixels to form a metal anti-corrosion gradient map comprises the following steps:

[0037] After the three-channel data after the color space numerical processing is aligned according to the time scale, the color space numerical fusion is performed;

[0038] Merge the 25,000,000*3 data set into 25,000,000 color pixels;

[0039] 25,000,000 pixels are stacked into 5000*5000 pixels to form a metal anti-corrosion gradient map.

[0040] In some embodiments, the training of a classification model based on the metal corrosion resistance gradient map and pre-classifying the metal corrosion resistance performance based on the classification model include the following steps:

[0041] According to the actual quality of metal samples, a preset number of metal samples are calibrated and divided into three grades;

[0042] Correspond the metal corrosion resistance gradient map of each sample to the metal corrosion resistance calibration;

[0043] Use the pre-trained ResNet image classification model for transfer learning, and replace the last layer of the pre-trained model with a model suitable for the classification output of the metal corrosion resistance gradient map using the train_model() function module;

[0044] Define an Incremental_Learning() function module, which is used to perform incremental learning, so as to continue to add calibration data in subsequent metal tests to the model for the model to learn, thereby completing continuous optimization of the classification model;

[0045] The metal corrosion resistance is pre-classified according to the classification model.

[0046] Another aspect of the embodiment of the present invention further provides an image representation device based on multimodal fusion, comprising:

[0047] The first module is used to obtain image data of the metal sample during the electrolysis process, calculate the total area of ​​the corroded part through edge detection and shape recognition, and process the obtained total area into first channel data;

[0048] The second module is used to measure the electrochemical impedance spectrum of the test metal, calculate the current density of the corroded metal using the corrosion current density formula combined with the electrochemical impedance spectrum, and process the obtained current density into the second channel data;

[0049] The third module is used to obtain and measure the acoustic spectrogram during the electrochemical corrosion test of the test metal, and convert the acoustic spectrogram into third channel data by weight allocation;

[0050] The fourth module is used to equally sample the first channel data, the second channel data and the third channel data to obtain RGB color space values;

[0051] A fifth module is used to merge the first channel data, the second channel data and the third channel data into a single RGB pixel point, and then stack the pixels to form a metal anti-corrosion gradient map;

[0052] The sixth module is used to obtain a classification model based on the metal corrosion resistance gradient map training, and pre-classify the metal corrosion resistance performance according to the classification model.

[0053] Another aspect of an embodiment of the present invention further provides an electronic device, including a processor and a memory;

[0054] The memory is used to store programs;

[0055] The processor executes the program to implement the method described above.

[0056] Another aspect of the embodiments of the present invention further provides a computer-readable storage medium, wherein the storage medium stores a program, and the program is executed by a processor to implement the method described above.

[0057] The embodiment of the present invention also discloses a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device can read the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the above method.

[0058] The embodiments of the present invention at least include the following beneficial effects: the present invention provides an image characterization method, device and electronic device based on multimodal fusion, the present invention obtains the image data of the metal sample in the electrolysis process, calculates the total area of ​​the corroded part by edge detection and shape recognition, and processes the obtained total area into the first channel data; measures the electrochemical impedance spectrum of the test metal, calculates the current density of the corroded metal by combining the corrosion current density formula with the electrochemical impedance spectrum, and processes the obtained current density into the second channel data; obtains and measures the spectrogram during the electrochemical corrosion test of the test metal, and converts the spectrogram into the third channel data by weight allocation; samples the first channel data, the second channel data and the third channel data equally to obtain the RGB color space value; fuses the first channel data, the second channel data and the third channel data into a single RGB pixel point, and then stacks the pixels to form a metal anti-corrosion gradient map; obtains a classification model based on the metal anti-corrosion gradient map, and pre-classifies the metal anti-corrosion performance based on the classification model. The present invention can realize the pre-classification of experimental metal samples, reduce unnecessary repeated experimental labor, and improve the efficiency of metal sample anti-corrosion experimental characterization. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 It is a schematic diagram of an implementation environment provided by an embodiment of the present invention;

[0060] Figure 2 is a flow chart of the overall steps provided by an embodiment of the present invention;

[0061] Figure 3 is a schematic diagram of pixels stacked according to an embodiment of the present invention;

[0062] Figure 4 1 is a schematic diagram of the implementation content of the Incremental_Learning() function module provided in an embodiment of the present invention;

[0063] Figure 5 It is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present invention. They are only examples of devices and methods consistent with some aspects of the embodiments of the present invention as detailed in the attached claims.

[0065] It is understood that the terms "first", "second", etc. used in the present invention may be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0066] The terms "at least one", "multiple", "each", "any", etc. used in the present invention, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0067] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.

[0068] The image representation method, device and electronic device based on multimodal fusion provided by the embodiment of the present invention relate to the field of image processing technology. The image representation method based on multimodal fusion provided by the embodiment of the present invention can be applied to a terminal, can be applied to a server, or can be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements an image representation method based on multimodal fusion, etc., but is not limited to the above forms.

[0069] The present invention can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0070] like Figure 1 FIG. 1 is a schematic diagram of an implementation environment provided by an embodiment of the present invention. Figure 1 The implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected to a network wirelessly or wired to complete data transmission and exchange.

[0071] Server 101 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0072] In addition, the server 101 can also be a node server in the blockchain network. Blockchain is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithm.

[0073] The terminal 102 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc. The terminal 102 may also be a vehicle-mounted terminal of various device types described above, but is not limited thereto. The terminal 102 and the server 101 may be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment of the present invention.

[0074] Based on the example Figure 1 In the implementation environment shown, an embodiment of the present invention provides an image representation method based on multimodal fusion. The following is explained using the application of the image representation method based on multimodal fusion in the server 101 as an example. It can be understood that the method can also be applied to the terminal 102.

[0075] Reference Figure 2 , Figure 2 The flowchart of the image representation method based on multimodal fusion applied to the server provided in the embodiment of the present invention, the execution subject of the method can be any of the aforementioned computer devices (including servers or terminals). Figure 2 , the method may include the following steps:

[0076] Acquire image data of the metal sample during the electrolysis process, calculate the total area of ​​the corroded part through edge detection and shape recognition, and process the obtained total area into first channel data;

[0077] Measure the electrochemical impedance spectrum of the test metal, calculate the current density of the corroded metal using the corrosion current density formula combined with the electrochemical impedance spectrum, and process the obtained current density into the second channel data;

[0078] Obtaining and measuring the acoustic spectrogram during the electrochemical corrosion test of the test metal, and converting the acoustic spectrogram into third channel data by weight allocation;

[0079] The first channel data, the second channel data and the third channel data are equally sampled to obtain RGB color space values;

[0080] The first channel data, the second channel data and the third channel data are merged into a single RGB pixel point, and then the pixel points are stacked to form a metal anti-corrosion gradient map;

[0081] A classification model is obtained by training the metal corrosion resistance gradient map, and the metal corrosion resistance performance is pre-classified according to the classification model.

[0082] Specifically, the present invention aims at the problem of low efficiency of metal electrochemical corrosion test process, and proposes a multimodal fusion three-channel composite image representation algorithm that integrates visual, electrochemical and auditory features, pre-classifies the metal samples in the experiment, and improves the efficiency of the corrosion test. Figure 2 The overall implementation steps shown in the figure include:

[0083] Step 1: Use two visual cameras to obtain image data of metal samples during the electrolysis process, and calculate the total area of ​​the corroded part through edge detection and shape recognition. Finally, the obtained area data is processed into the first channel data of corrosion area-time.

[0084] Step 2: Measure the electrochemical impedance spectrum of the test metal through a multi-channel electrochemical workstation, and calculate the current density of the corroded metal using the corrosion current density formula combined with the electrochemical impedance spectrum. Finally, process the obtained current density data into the second channel data such as current density-time.

[0085] Step 3: The spectrum data of the electrochemical corrosion test of the test metal is measured by the spectrum analyzer, and the spectrum is converted into the third channel data of corrosion intensity-time by weight allocation.

[0086] Step 4: Sample the data of the three channels 25,000,000 times in equal amounts, normalize the sampled data and process them into RGB color space values.

[0087] Step 5: The 25000000*3 structured data of the three channels are merged into a single RGB pixel, and then the pixels are stacked to form a metal anti-corrosion gradient map.

[0088] Step 6: Classify the metal corrosion resistance into the first, second and third grades. The metal corrosion resistance is pre-classified into three grades according to the classification model trained with the metal corrosion resistance gradient map corresponding to the corrosion resistance data measured and calibrated manually in advance.

[0089] In some embodiments, the step 1 uses two visual cameras to obtain electrochemical corrosion images of two diagonal angles of the metal block sample. Then, each image obtained is binarized using a threshold binarization algorithm. Then, findContours() and drawContours() functions in opencv are used to perform edge detection and contour processing. Then, the cv.contourArea() function is used to solve the contour area.

[0090] Since only two visual cameras are used to detect the corrosion area of ​​the metal sample, the angle between the camera and the plane of the block metal is 45°, so after using the above algorithm to solve the corrosion area S(t), the actual corrosion area is 2*S(t). S(t) represents the change of the corrosion area with the electrochemical test time, which is the first channel data.

[0091] In some embodiments, the step 2 uses a multi-channel electrochemical workstation to obtain an electrochemical impedance spectrum (EIS) of the corroded metal, and uses a Randle's circuit to fit the EIS data to obtain the electrolyte resistance R s , double layer capacitance C dl and polarization resistance R p According to the fitting polarization resistance R p The Stern-Geary constant B calculated by formula (1.1) is used to calculate the corrosion current density j using formula (1.2) corr .

[0092]

[0093] Among them, β a is the anodic Tafel slope, β c is the cathodic Tafel slope.

[0094]

[0095] Among them, j corr is the corrosion current density, R p is the polarization resistance and B is the Stern-Geary constant.

[0096] Then the calculated second channel data is j corr (t).

[0097] In some embodiments, the step 3 receives and measures the spectrum data of the test metal electrochemical corrosion test process through a spectrum analyzer. The spectrum data contains two data: frequency and amplitude. We use the weight distribution of frequency 0.05 and amplitude 0.95 to perform linear weighted superposition processing on the spectrum data. The data obtained after processing is calibrated as corrosion intensity Q(t), which is the third channel data.

[0098] In some embodiments, in step 4, 25,000,000 data samples are collected for the corrosion test of each group of metal samples. After processing the data of the three channels through steps 1, 2, and 3, the data set S(t) is obtained. corr (t) and Q(t). Then, we use formula (1.3) on the above three data sets respectively to perform RGB color space numerical processing on the three-channel data.

[0099]

[0100] Among them, x1(t) represents S(t) after numerical processing in RGB color space, j corr (t) and Q(t) data, x(t) represents the original data set S(t), j corr (t) and Q(t), the subscripts min and max represent the minimum and maximum values ​​of the current data set, respectively.

[0101] In some embodiments, the step 5 aligns the three-channel data after the color space digitization processing in step 4 according to the time scale, and then performs the color space numerical fusion. The 25000000*3 data set is fused into 25000000 color pixels. Then these 25000000 pixels are stacked into 5000*5000 pixels. The stacking method is as follows: Figure 3 shown.

[0102] In some embodiments, the step 6 first manually calibrates a certain number of metal samples according to the actual quality of the metal samples, and divides them into three grades, and the metal corrosion resistance gradient map of each sample generated in step 5 corresponds to the metal corrosion resistance calibration. First, we use the pre-trained ResNet image classification model for transfer learning, and replace the last layer of the pre-trained model with a model suitable for the classification output of the metal corrosion resistance gradient map using the train_modell() function module. Then define the Incremental_Learning() function module to realize the subsequent incremental learning function, that is, the calibration data in the subsequent metal test can continue to be added to the model for the model to learn and continuously optimize the classification model.

[0103] Regarding the definition of Incremental_Learning(), such as Figure 4 As shown, the Incremental_Learning() function module of the embodiment of the present invention can be used to obtain new training data and verification sets, define loss functions and optimizers, fine-tune models, and verify models.

[0104] Assume that there are initial datasets in 'path_to_initial_train_data' and 'path_to_initial_val_data', and new datasets in 'path_to_new_train_data' and 'path_to_val_data'. You can add new 'path_to_new_train_data' and 'path_to_val_data' datasets to the pre-trained model to optimize the previous model to achieve incremental learning.

[0105] In summary, the present invention is reasonably designed. The multimodal fusion three-channel composite image characterization algorithm model that integrates visual, electrochemical and auditory features can effectively improve the characterization efficiency of metal corrosion resistance tests in the process of high-throughput preparation and characterization of metal materials. (Through pre-classification, professionals can subdivide from high-quality categories when analyzing the corrosion resistance of metals to reduce labor costs) It reduces the input of labor costs and improves the level of characterization automation. At the same time, because the classification is carried out by integrating multiple modalities, the classification effect is significant, which reduces the situation in which excellent material samples are easily ignored due to one-sided classification discrimination features in single-modal classification algorithms.

[0106] The embodiment of the present invention further provides an image representation device based on multimodal fusion, comprising:

[0107] The first module is used to obtain image data of the metal sample during the electrolysis process, calculate the total area of ​​the corroded part through edge detection and shape recognition, and process the obtained total area into first channel data;

[0108] The second module is used to measure the electrochemical impedance spectrum of the test metal, calculate the current density of the corroded metal using the corrosion current density formula combined with the electrochemical impedance spectrum, and process the obtained current density into the second channel data;

[0109] The third module is used to obtain and measure the acoustic spectrogram during the electrochemical corrosion test of the test metal, and convert the acoustic spectrogram into third channel data by weight allocation;

[0110] The fourth module is used to equally sample the first channel data, the second channel data and the third channel data to obtain RGB color space values;

[0111] A fifth module is used to merge the first channel data, the second channel data and the third channel data into a single RGB pixel point, and then stack the pixels to form a metal anti-corrosion gradient map;

[0112] The sixth module is used to obtain a classification model based on the metal corrosion resistance gradient map training, and pre-classify the metal corrosion resistance performance according to the classification model.

[0113] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0114] The embodiment of the present invention further provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned image representation method based on multimodal fusion when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0115] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0116] See also Figure 5 , Figure 5 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0117] The processor 501 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.

[0118] The memory 502 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 502 can store an operating system and other application programs. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 502, and the processor 501 calls and executes the image representation method based on multimodal fusion in the embodiment of the present invention;

[0119] Input / output interface 503, used to implement information input and output;

[0120] Communication interface 504, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WI FI, Bluetooth, etc.);

[0121] A bus 505 that transmits information between the various components of the device (e.g., the processor 501, the memory 502, the input / output interface 503, and the communication interface 504);

[0122] The processor 501 , the memory 502 , the input / output interface 503 and the communication interface 504 are connected to each other in communication within the device via the bus 505 .

[0123] An embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the image representation method based on multimodal fusion is implemented.

[0124] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0125] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0126] The embodiments described in the embodiments of the present invention are intended to more clearly illustrate the technical solutions of the embodiments of the present invention, and do not constitute a limitation on the technical solutions provided by the embodiments of the present invention. Those skilled in the art can appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present invention are also applicable to similar technical problems.

[0127] Those skilled in the art will appreciate that the technical solutions shown in the figures do not limit the embodiments of the present invention and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0128] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0129] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0130] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0131] It should be understood that in the present invention, "at least one (item)" refers to one or more, and "plurality" refers to two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can represent: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0132] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0133] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0134] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0135] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store programs.

[0136] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the embodiments of the present invention is not limited thereby. Any modification, equivalent substitution and improvement made by those skilled in the art without departing from the scope and essence of the embodiments of the present invention shall be within the scope of the rights of the embodiments of the present invention.

Claims

1. An image representation method based on multimodal fusion, characterized in that: The following steps are involved: Acquire image data of the metal sample during the electrolysis process, calculate the total area of ​​the corroded part through edge detection and shape recognition, and process the obtained total area into first channel data; Measure the electrochemical impedance spectrum of the test metal, calculate the current density of the corroded metal using the corrosion current density formula combined with the electrochemical impedance spectrum, and process the obtained current density into the second channel data; Obtaining and measuring the acoustic spectrogram during the electrochemical corrosion test of the test metal, and converting the acoustic spectrogram into third channel data by weight allocation; The first channel data, the second channel data and the third channel data are equally sampled to obtain RGB color space values; The first channel data, the second channel data and the third channel data are merged into a single RGB pixel point, and then the pixel points are stacked to form a metal anti-corrosion gradient map; A classification model is obtained by training the metal corrosion resistance gradient map, and the metal corrosion resistance performance is pre-classified according to the classification model.

2. The image representation method based on multimodal fusion according to claim 1, characterized in that: The method of obtaining image data of a metal sample during electrolysis, calculating the total area of ​​the corroded portion by edge detection and shape recognition, and processing the obtained total area into first channel data comprises the following steps: Using two visual cameras, electrochemical corrosion images of metal block samples at two diagonal angles are obtained; Each electrochemical corrosion image obtained is binarized using the threshold binarization algorithm; Use findContours() and drawContours() functions in opencv for edge detection and contour processing; Use cv2.contourArea() function to solve the contour area and calculate the total area of ​​the corroded part; The first channel data is processed into corrosion area-time according to the total area of ​​the corroded part.

3. The image representation method based on multimodal fusion according to claim 1, characterized in that: The method of measuring the electrochemical impedance spectrum of the test metal, calculating the current density of the corroded metal using the corrosion current density formula in combination with the electrochemical impedance spectrum, and processing the obtained current density into second channel data includes the following steps: The electrochemical impedance spectrum of the corroded metal was obtained using a multi-channel electrochemical workstation, and the electrochemical impedance spectrum data was fitted using Randle's circuit to obtain the electrolyte resistance R s , double layer capacitance C dl and polarization resistance R p ; The calculated Stern-Geary constant B; According to the fitting polarization resistance R p And according to the Stern-Geary constant B, the corrosion current density j is calculated corr ; Among them, the calculation formula of Stern-Geary constant B is: Corrosion current density j corr The calculation formula is: Among them, β a is the anodic Tafel slope, β c is the cathode Tafel slope; j corr is the corrosion current density; R p is the polarization resistance; B is the Stern-Geary constant.

4. The image representation method based on multimodal fusion according to claim 1, characterized in that: The method of obtaining and measuring the acoustic spectrogram during the electrochemical corrosion test of the test metal and converting the acoustic spectrogram into third channel data by weight allocation comprises the following steps: The spectrum data of the test metal electrochemical corrosion test is received and measured by the spectrum analyzer; wherein the spectrum data includes two data: frequency and amplitude; The acoustic spectrum data is processed by linear weighted superposition using a weight distribution of frequency 0.05 and amplitude 0.

95. The data obtained after processing is calibrated as corrosion intensity and is the third channel data.

5. The image representation method based on multimodal fusion according to claim 1, characterized in that: The step of sampling the first channel data, the second channel data, and the third channel data equally to obtain RGB color space values ​​comprises the following steps: 25,000,000 data samples were collected for each group of metal sample corrosion tests; The first channel data, the second channel data and the third channel data are numerically processed in RGB color space to obtain RGB color space values. The expression of the processing process is: Among them, x1(t) represents the first channel data, the second channel data and the third channel data after numerical processing in RGB color space, x(t) represents the original first channel data, the second channel data and the third channel data, and the subscripts min and max represent the minimum and maximum values ​​of the current data set, respectively.

6. The image representation method based on multimodal fusion according to claim 1, characterized in that: The step of fusing the first channel data, the second channel data and the third channel data into a single RGB pixel point, and then stacking the pixels to form a metal anti-corrosion gradient map comprises the following steps: After the three-channel data after the color space numerical processing is aligned according to the time scale, the color space numerical fusion is performed; Merge the 25,000,000*3 data set into 25,000,000 color pixels; 25,000,000 pixels are stacked into 5000*5000 pixels to form a metal anti-corrosion gradient map.

7. The image representation method based on multimodal fusion according to claim 1, characterized in that: The step of obtaining a classification model according to the metal corrosion resistance gradient map training and pre-classifying the metal corrosion resistance performance according to the classification model comprises the following steps: According to the actual quality of metal samples, a preset number of metal samples are calibrated and divided into three grades; Correspond the metal corrosion resistance gradient map of each sample to the metal corrosion resistance calibration; Use the pre-trained ResNet image classification model for transfer learning, and replace the last layer of the pre-trained model with a model suitable for the classification output of the metal corrosion resistance gradient map using the train_model() function module; Define an Incremental_Learning() function module, which is used to perform incremental learning, so as to continue to add calibration data in subsequent metal tests to the model for the model to learn, thereby completing continuous optimization of the classification model; The metal corrosion resistance is pre-classified according to the classification model.

8. An image representation device based on multimodal fusion, characterized in that: include: The first module is used to obtain image data of the metal sample during the electrolysis process, calculate the total area of ​​the corroded part through edge detection and shape recognition, and process the obtained total area into first channel data; The second module is used to measure the electrochemical impedance spectrum of the test metal, calculate the current density of the corroded metal using the corrosion current density formula combined with the electrochemical impedance spectrum, and process the obtained current density into the second channel data; The third module is used to obtain and measure the acoustic spectrogram during the electrochemical corrosion test of the test metal, and convert the acoustic spectrogram into third channel data by weight allocation; The fourth module is used to equally sample the first channel data, the second channel data and the third channel data to obtain RGB color space values; A fifth module is used to merge the first channel data, the second channel data and the third channel data into a single RGB pixel point, and then stack the pixels to form a metal anti-corrosion gradient map; The sixth module is used to obtain a classification model based on the metal corrosion resistance gradient map training, and pre-classify the metal corrosion resistance performance according to the classification model.

9. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores a program, and the program is executed by a processor to implement the method according to any one of claims 1 to 7.

Citation Information

Patent Citations

  • Multi-modal data fusion lane target detection method and system based on multi-scale convolution

    CN113569803A

  • Propeller cavitation corrosion area monitoring method and system and acquisition device

    CN117788380A

  • Method for measuring amount of hydrogen penetrating inside metal and method for monitoring amount of hydrogen penetrating inside metal site of movable body

    JP2014089207A

Cited By

  • Battery health state estimation method and system based on electrochemical impedance spectroscopy

    CN122410370A