Multi-modal data processing method

Abstract

The invention relates to the technical field of data processing methods, in particular to a multi-modal data processing method which comprises the following steps: S1, a data collection module receives and processes related data, obtains information of mathematical scores and physical scores and transmits the information to a control module; s2, the control module calculates sorted mathematical scores according to the mathematical scores, calculates sorted physical scores according to the physical scores, calculates correlation factors of the mathematical scores and the physical scores, obtains correlation information of the mathematical scores and the physical scores according to the correlation factors of the mathematical scores and the physical scores, and transmits the correlation information to a communication module; and S3, the communication module transmits the relevance information of the mathematical score and the physical score to a user side. Whether two groups of data have relevance or not can be quickly obtained through the control module, and the model training speed is increased.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing methods, and in particular to a multimodal data processing method. Background Art

[0002] Multimodality refers to the collaborative reasoning of multiple heterogeneous modal data. It integrates or fuses two or more biometric technologies, utilizes the unique advantages of multiple biometric technologies, and combines data fusion technology to make the authentication and identification process more accurate and secure.

[0003] In the field of artificial intelligence, multimodality usually refers to the use of multiple information input methods, such as images, text, and voice, to help artificial intelligence understand the external world more accurately. In daily life, we can also express our meaning through multimodality, such as through language, gestures, facial expressions, etc. to convey our emotions and intentions.

[0004] Association model building mainly involves associating multiple data tables to discover the relationships between them.

[0005] After extensive searching and reference, we found that many training methods for association construction models have been developed. For example, the prior art method for training association construction models disclosed in publication number CN115934688A includes the following steps: data collection: collecting original data related to the association model subject; data cleaning: processing and analyzing the collected data to eliminate errors, outliers and duplicate information; feature extraction: extracting features related to the association model from the data; model training: training the association model based on the extracted features; model prediction: using the trained association model to make predictions or decisions.

[0006] The above training process is long and complicated. When users urgently need to obtain the results of an association construction model, the existing technology cannot meet their needs. Summary of the invention

[0007] The purpose of the present invention is to improve the processing speed. In view of the above-mentioned shortcomings, a multimodal data processing method is proposed.

[0008] The present invention adopts the following technical solution:

[0009] A multimodal data processing method comprises the following steps:

[0010] S1: The data collection module receives and processes relevant data and obtains information on math scores and physics scores, and transmits it to the control module;

[0011] S2: The control module calculates the sorted math scores according to the math scores, calculates the sorted physics scores according to the physics scores, calculates the correlation factor between the math scores and the physics scores according to the covariance of the sorted math scores and the sorted physics scores, the standard deviation of the sorted math scores, and the standard deviation of the sorted physics scores, and obtains the correlation information between the math scores and the physics scores according to the correlation factor between the math scores and the physics scores and transmits it to the communication module;

[0012] S3: The communication module transmits the correlation information between the mathematics scores and the physics scores to the user end.

[0013] Optionally, the data collection module includes a multimodal data input submodule, a data conversion submodule and a data integration submodule, the data conversion submodule and the data integration submodule are both communicatively connected to the multimodal data input submodule, and the data integration submodule is communicatively connected to the control module;

[0014] The multimodal data input submodule is used to receive relevant data of image, text and voice input, transmit relevant data of image and voice input to the data conversion submodule, and transmit relevant data of text input to the data integration submodule;

[0015] The data conversion submodule is used to convert the relevant data of the image and voice input, and transmit the converted relevant data to the data integration submodule;

[0016] The data integration submodule integrates the text input related data and the converted related data, obtains the information of the mathematics score and the physics score, and transmits it to the control module.

[0017] Optionally, when the control module calculates the correlation factor between the math score and the physics score, the following formula is satisfied:

[0018]

[0019] X=[x n , x n+1 , …, x N ];

[0020] Y=[y n ,y n+1 , …, y N ];

[0021] ΔX=[x n *α,x n+1 *α,…,x N *α];

[0022] ΔY=[y n *α,y n+1 *α,…,y N*α];

[0023] Among them, GLX is the correlation factor between mathematics scores and physics scores, XFC(ΔX,ΔY) is the covariance of the adjusted mathematics scores and the adjusted physics scores, BZC ΔX is the standard deviation of the math scores after adjustment, BZC ΔY is the standard deviation of the adjusted physics scores, ΔX is the adjusted mathematics scores, and ΔY is the adjusted physics scores;

[0024] X is the math score, x n is the math score of the nth student, x N is the math score of the Nth student;

[0025] Y is the physics score, y n is the physics score of the nth student, y N is the physics score of the Nth student;

[0026] α is a conversion index, and α has the following values: α=1 or α=0.98 or α=0.95. When α=1, the student's score is input via text; when α=0.98, the student's score is input via voice; and when α=0.95, the student's score is input via image.

[0027] Optionally, when the control module calculates the correlation information between the math scores and the physics scores, the following formula is satisfied:

[0028]

[0029] Among them, β is the correlation information between mathematics scores and physics scores, γ1 to γ4 are the selection thresholds of different correlation factors between mathematics scores and physics scores, when β = 1, it indicates that there is a negative linear relationship between mathematics scores and physics scores, when β = 2, it indicates that there is no linear relationship between mathematics scores and physics scores, and when β = 3, it indicates that there is a positive linear relationship between mathematics scores and physics scores.

[0030] Optionally, in step S1, the following steps are also included:

[0031] The data collection module transmits the information of mathematics scores and physics scores to the data cleaning module;

[0032] The data cleaning module deletes abnormal data and missing data from the math scores, and obtains information on the total number of math scores before data cleaning, the math score of the z-th student after data cleaning, the total number of math scores after data cleaning, the total duration of data cleaning, and the math score of the Z-th student after data cleaning, and transmits the information to the control module;

[0033] In step S2, the following steps are also included:

[0034] The control module calculates the math score after data cleaning according to the math score of the Zth student after data cleaning and the math score of the zth student after data cleaning, calculates the data cleaning determination factor according to the total number of math scores before data cleaning, the math score of the nth student, the total number of math scores after data cleaning, and the math score of the zth student after data cleaning, and obtains alarm information according to the data cleaning determination factor and transmits it to the communication module;

[0035] In step S3, the following steps are also included:

[0036] The communication module transmits the alarm information to the user end and the alarm end.

[0037] Optionally, the data cleaning module includes a data cleaning submodule and a data analysis submodule that are communicatively connected, and the data analysis submodule is communicatively connected to the control module;

[0038] The data cleaning submodule cleans the abnormal data and missing data of the mathematics scores and obtains the relevant data after the data cleaning, and transmits it to the data analysis submodule;

[0039] The data analysis submodule is used to analyze and obtain information on the total number of students with math scores before data cleaning, the math score of the zth student after data cleaning, the total number of students with math scores after data cleaning, the total duration of data cleaning, and the math score of the Zth student after data cleaning, and transmit it to the control module.

[0040] Optionally, the communication module includes an information transmission submodule and an alarm submodule that are communicatively connected, and the information transmission submodule and the alarm submodule are both communicatively connected to the control module;

[0041] The control module transmits the correlation information between the math scores and the physics scores to the information transmission submodule, and synchronously transmits the alarm information to the information transmission submodule and the alarm submodule;

[0042] The information transmission submodule transmits the correlation information of the mathematics scores and the physics scores and the alarm information to the user end;

[0043] The alarm submodule transmits the alarm information to the alarm terminal and implements the alarm according to the alarm information.

[0044] Optionally, when the control module calculates the data cleaning determination factor, the following formula is satisfied:

[0045]

[0046] X c =[x z , xz+1 , …, x Z ];

[0047] Among them, QX is the data cleaning determination factor, N is the total number of math scores before data cleaning, and x z is the math score of the zth student after data cleaning, Z is the total number of students with math scores after data cleaning, and T is the total time of data cleaning;

[0048] X c is the math score after data cleaning, x Z This is the math score of the Zth student after data cleaning.

[0049] Optionally, when the control module calculates the alarm information, the following formula is satisfied:

[0050]

[0051] Among them, BJ is the alarm information, qx ref It is the selection threshold of the data cleaning judgment factor. When BJ=1, the data cleaning is normal and no alarm is needed. When BJ=2, the data cleaning is abnormal and an alarm is needed.

[0052] The beneficial effects achieved by the present invention are:

[0053] 1. Through the control module, it is possible to quickly determine whether two sets of data are related. That is, calculating the correlation information between math scores and physics scores is equivalent to quickly confirming the correlation between the two sets of data, which is convenient for subsequent model training operations and speeds up model training;

[0054] 2. Since the data cleaning in the prior art does not set relevant cleaning requirements, after the data cleaning step, the control module calculates the alarm information to further determine whether the data cleaning is abnormal, which can improve the accuracy of the model building;

[0055] 3. The alarm submodule can give real-time alarms based on the alarm information, thereby speeding up the overall response speed and enabling staff to promptly discover abnormal data cleaning problems.

[0056] To further understand the features and technical contents of the present invention, please refer to the following detailed description and drawings of the present invention. However, the drawings provided are only for reference and description and are not intended to limit the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 is a flow chart of the method of the present invention;

[0058] Figure 2 It is a schematic diagram of the overall structure of the present invention;

[0059] Figure 3 It is a schematic diagram of the overall structure of the second embodiment of the present invention. DETAILED DESCRIPTION

[0060] The following is an explanation of the embodiments of the present invention through specific embodiments. Those skilled in the art can understand the advantages and effects of the present invention from the contents disclosed in this specification. The present invention can be implemented or applied through other different specific embodiments, and the details in this specification can also be modified and changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. In addition, the drawings of the present invention are only simple schematic illustrations and are not depicted according to actual dimensions. It is stated in advance. The following embodiments will further explain the relevant technical contents of the present invention in detail, but the disclosed contents are not intended to limit the scope of protection of the present invention.

[0061] Embodiment 1: This embodiment provides a multimodal data processing method, combining Figure 1 and Figure 2 shown.

[0062] A multimodal data processing method comprises the following steps:

[0063] S1: The data collection module receives and processes relevant data and obtains information on math scores and physics scores, and transmits it to the control module;

[0064] S2: The control module calculates the sorted math scores according to the math scores, calculates the sorted physics scores according to the physics scores, calculates the correlation factor between the math scores and the physics scores according to the covariance of the sorted math scores and the sorted physics scores, the standard deviation of the sorted math scores, and the standard deviation of the sorted physics scores, and obtains the correlation information between the math scores and the physics scores according to the correlation factor between the math scores and the physics scores and transmits it to the communication module;

[0065] S3: The communication module transmits the correlation information between the mathematics scores and the physics scores to the user end.

[0066] Optionally, the data collection module includes a multimodal data input submodule, a data conversion submodule and a data integration submodule, the data conversion submodule and the data integration submodule are both communicatively connected to the multimodal data input submodule, and the data integration submodule is communicatively connected to the control module;

[0067] The multimodal data input submodule is used to receive relevant data of image, text and voice input, transmit relevant data of image and voice input to the data conversion submodule, and transmit relevant data of text input to the data integration submodule;

[0068] The data conversion submodule is used to convert the relevant data of the image and voice input, and transmit the converted relevant data to the data integration submodule;

[0069] The data integration submodule integrates the text input related data and the converted related data, obtains the information of the mathematics score and the physics score, and transmits it to the control module.

[0070] Optionally, when the control module calculates the correlation factor between the math score and the physics score, the following formula is satisfied:

[0071]

[0072] X=[x n , x n+1 , …, x N ];

[0073] Y=[y n ,y n+1 , …, y N ];

[0074] ΔX=[x n *α,x n+1 *α,…,x N *α];

[0075] ΔY=[y n *α,y n+1 *α,…,y N *α];

[0076] Among them, GLX is the correlation factor between mathematics scores and physics scores, XFC(ΔX,ΔY) is the covariance of the adjusted mathematics scores and the adjusted physics scores, BZC ΔX is the standard deviation of the math scores after adjustment, BZC ΔY is the standard deviation of the adjusted physics scores, ΔX is the adjusted mathematics scores, and ΔY is the adjusted physics scores;

[0077] X is the math score, x n is the math score of the nth student, x N is the math score of the Nth student;

[0078] Y is the physics score, y n is the physics score of the nth student, y N is the physics score of the Nth student;

[0079] α is a conversion index, and α has the following values: α=1 or α=0.98 or α=0.95. When α=1, the student's score is input via text; when α=0.98, the student's score is input via voice; and when α=0.95, the student's score is input via image.

[0080] Specifically, the following matters need to be noted when calculating the conversion index: inputting students' scores via text means using text input when entering the data input system, that is, the system directly outputs students' scores during online grading, and students' scores are input via text at this time; inputting students' scores via voice means that teachers directly input students' scores via voice when entering the data input system, and voice needs to be converted into text at this time; inputting students' scores via images means that teachers take a picture of the test scores when entering the data input system, and images need to be converted into text at this time. Since there are certain errors in the way voice and image are converted into text, setting the conversion index is equivalent to the accuracy of the conversion.

[0081] The following specific numbers are used to illustrate how to calculate the correlation factors of math scores and physics scores. If the math scores are set to "80, 85, 90, 75, 100" and the physics scores are set to "70, 75, 85, 60, 90", the corresponding math scores and physics formulas are:

[0082] X = [80, 85, 90, 75, 100];

[0083] Y = [70,75,85,60,90];

[0084] Suppose the score of the fifth student in the math score is input by image, the scores of the remaining students are input by text, and the scores of all students in the physics score are input by text. The corresponding formulas of the sorted math score and the sorted physics score are:

[0085] ΔX=[80*1,85*1,90*1,75*1,100*0.95]=[80,85,90,75,95];

[0086] ΔY=[70*1,75*1,85*1,60*1,90*1]=[70,75,85,60,90];

[0087] Again

[0088] Again

[0089] Then calculate the covariance of the sorted math scores and the sorted physics scores, which is the following formula:

[0090]

[0091] Then calculate the standard deviation of the math scores after sorting, which is the following formula:

[0092]

[0093] Then calculate the standard deviation of the physics scores after sorting, which is the following formula:

[0094]

[0095] Then calculate the correlation factor between mathematics scores and physics scores, which is the following formula:

[0096]

[0097] Optionally, when the control module calculates the correlation information between the math scores and the physics scores, the following formula is satisfied:

[0098]

[0099] Among them, β is the correlation information between mathematics scores and physics scores, γ1 to γ4 are the selection thresholds of different correlation factors between mathematics scores and physics scores, when β = 1, it indicates that there is a negative linear relationship between mathematics scores and physics scores, when β = 2, it indicates that there is no linear relationship between mathematics scores and physics scores, and when β = 3, it indicates that there is a positive linear relationship between mathematics scores and physics scores.

[0100] Preferably, specific numbers are listed below to illustrate how to calculate the correlation information between math scores and physics scores, such as the following formula:

[0101]

[0102] It can be further set that when the correlation factor between math scores and physics scores is closer to -1.25, the negative linear relationship is more obvious, and when the correlation factor between math scores and physics scores is closer to 0.75, the positive linear relationship is more obvious. When applied in other embodiments, the selection threshold of the corresponding correlation factor between math scores and physics scores will be appropriately changed.

[0103] Optionally, in step S1, the following steps are also included:

[0104] The data collection module transmits the information of mathematics scores and physics scores to the data cleaning module;

[0105] The data cleaning module deletes abnormal data and missing data from the math scores, and obtains information on the total number of math scores before data cleaning, the math score of the z-th student after data cleaning, the total number of math scores after data cleaning, the total duration of data cleaning, and the math score of the Z-th student after data cleaning, and transmits the information to the control module;

[0106] In step S2, the following steps are also included:

[0107] The control module calculates the math score after data cleaning according to the math score of the Zth student after data cleaning and the math score of the zth student after data cleaning, calculates the data cleaning determination factor according to the total number of math scores before data cleaning, the math score of the nth student, the total number of math scores after data cleaning, and the math score of the zth student after data cleaning, and obtains alarm information according to the data cleaning determination factor and transmits it to the communication module;

[0108] In step S3, the following steps are also included:

[0109] The communication module transmits the alarm information to the user end and the alarm end.

[0110] Optionally, the data cleaning module includes a data cleaning submodule and a data analysis submodule that are communicatively connected, and the data analysis submodule is communicatively connected to the control module;

[0111] The data cleaning submodule cleans the abnormal data and missing data of the mathematics scores and obtains the relevant data after the data cleaning, and transmits it to the data analysis submodule;

[0112] The data analysis submodule is used to analyze and obtain information on the total number of students with the best math scores before data cleaning, the math scores of the zth student after data cleaning, the total number of students with the best math scores after data cleaning, the total duration of data cleaning, and the math scores of the Zth student after data cleaning, and transmit it to the control module.

[0113] Optionally, the communication module includes an information transmission submodule and an alarm submodule that are communicatively connected, and the information transmission submodule and the alarm submodule are both communicatively connected to the control module;

[0114] The control module transmits the correlation information between the math scores and the physics scores to the information transmission submodule, and synchronously transmits the alarm information to the information transmission submodule and the alarm submodule;

[0115] The information transmission submodule transmits the correlation information of the mathematics scores and the physics scores and the alarm information to the user end;

[0116] The alarm submodule transmits the alarm information to the alarm terminal and implements the alarm according to the alarm information.

[0117] Optionally, when the control module calculates the data cleaning determination factor, the following formula is satisfied:

[0118]

[0119] X c =[x z , x z+1 , …, x Z ];

[0120] Among them, QX is the data cleaning determination factor, N is the total number of math scores before data cleaning, and x z is the math score of the zth student after data cleaning, Z is the total number of students with math scores after data cleaning, and T is the total time of data cleaning;

[0121] X c is the math score after data cleaning, x z This is the math score of the Zth student after data cleaning.

[0122] Specifically, data cleaning refers to deleting abnormal data and missing data. Abnormal data refers to data that is extreme or unreasonable, and missing data refers to data that is not filled in during input. The following uses specific values ​​to illustrate how to delete abnormal data. When entering math scores, the full score is known to be 120, a's math score is 50, b's math score is 60, c's math score is not entered, d's math score is 80, e's math score is 130, and f's math score is D. The above integration in the formula is as follows:

[0123] X = [50, 60, not entered, 80, 130, grade D];

[0124] The corresponding "not entered" is classified as missing data, and "130" and "Grade D" are abnormal data, which are integrated into the formula as follows:

[0125] X c = [50, 60, 80];

[0126] All the classified missing data and abnormal data will be deleted.

[0127] Optionally, when the control module calculates the alarm information, the following formula is satisfied:

[0128]

[0129] Among them, BJ is the alarm information, qx ref It is the selection threshold of the data cleaning judgment factor. When BJ=1, the data cleaning is normal and no alarm is needed. When BJ=2, the data cleaning is abnormal and an alarm is needed.

[0130] Although no specific units are provided for the above formula in this embodiment, those skilled in the art will certainly set appropriate units according to the needs of actual applications.

[0131] This embodiment solves the problem of long processing time in the traditional training method of building a multimodal association model. Specifically, this embodiment can quickly determine whether two sets of data are correlated through the control module, which facilitates subsequent model training operations and speeds up model training.

[0132] In addition, since the data cleaning in the prior art does not set relevant cleaning requirements, after the data cleaning step, the control module calculates the alarm information to further determine whether the data cleaning is abnormal, which can improve the accuracy of the constructed model.

[0133] Finally, the alarm submodule can issue an alarm in real time based on the alarm information, thereby speeding up the overall response speed and enabling staff to promptly discover abnormal data cleaning problems.

[0134] Embodiment 2: This embodiment includes all the contents of embodiment 1, and provides a multimodal data processing method, combined with Figure 3 shown.

[0135] A multimodal data processing method further includes the following steps:

[0136] In step S3, the following steps are also included:

[0137] The communication module transmits the alarm information to the abnormal data acquisition module.

[0138] The abnormal data acquisition module identifies abnormal data and statistical information, and obtains the total number of historical scores, the cth historical score, the Cth historical score, and the student's leave reference index for this semester, and transmits the information to the control module;

[0139] In step S2, the following steps are also included:

[0140] The control module calculates the calculated value of the abnormal data according to the total number of historical scores, the cth historical score, the Cth historical score, and the student's leave reference index this semester, and obtains the true value of the abnormal data based on the calculated value of the abnormal data and transmits it to the data collection module.

[0141] Specifically, when the alarm submodule implements an alarm, it indicates that the data cleaning is abnormal, that is, too much data is deleted, resulting in the overall data being not of reference value. Therefore, by adding an abnormal data acquisition module and a control module to cooperate, the true value of the abnormal data is calculated, and the true value of the abnormal data is added to the data integration submodule, and then related operations are performed, which will reduce the amount of deleted data; setting the true value of the abnormal data calculated by the control module belongs to the text input method.

[0142] Optionally, when the control module calculates the calculated value of the abnormal data, the following formula is satisfied:

[0143]

[0144] Among them, YCSJ js is the calculated value of abnormal data, C is the total number of historical scores, CJ cFor the cth historical result, CJ C is the Cth historical score, CQ is the reference index of students' leave this semester, and CQ has the following values: CQ=0.95 or CQ=0.90 or CQ=0.85. When CQ=0.95, the cumulative number of days of leave for students this semester is less than 8 days; when CQ=0.90, the cumulative number of days of leave for students this semester ranges from greater than or equal to 8 days to less than 15 days; when CQ=0.85, the cumulative number of days of leave for students this semester ranges from greater than or equal to 15 days.

[0145] Specifically, the following matters need to be noted when calculating the total number of historical scores. The historical score refers to the previous C historical scores taken as reference scores when the current score is abnormal, and the historical scores and the current scores are set to be the same subject. In addition, the total number of historical scores is limited to greater than or equal to 5 times, that is, C≥5. When C<5, the abnormal data is converted into missing data and the missing data is deleted.

[0146] Optionally, when the control module calculates the true value of the abnormal data, the following formula is satisfied:

[0147]

[0148] Among them, YCSJ tc is the true value of the abnormal data, and μ is the selection threshold of the score difference.

[0149] Specifically, the selection threshold of the performance difference is preset by those skilled in the art.

[0150] Although no specific units are provided for the above formula in this embodiment, those skilled in the art will certainly set appropriate units according to the needs of actual applications.

[0151] After the calculation value of the above-mentioned abnormal data is completed, the existing association construction model is used to analyze the correlation between the scores of different subjects, and the Spearman rank correlation coefficient is specifically used to measure the dependence between two variables. The analysis principle is: first, each variable is assigned a grade, and then the Spearman rank correlation coefficient is calculated using the formula of the Spearman correlation coefficient. For example, taking the calculation of the correlation between mathematics scores and physics scores as an example, first, the grades of mathematics and physics are assigned grades, and the students with the highest grades are limited to get grade 1. The lower the grade, the greater the value of the grade. Then, the Spearman rank correlation coefficient is calculated according to the above steps. When the grade of one of the students is lost, the grade ranking of this student cannot be calculated, so the data of this student needs to be excluded during calculation. In the prior art, a related algorithm is required to rank the grades, which is relatively cumbersome, and when the rankings of students are the same, it is easy to make mistakes, which will slow down the overall calculation speed. Therefore, the present invention estimates the correlation through a simple calculation method, and the user can quickly estimate the correlation. If the user needs to accurately know the correlation degree of the correlation, it can be judged whether the cleaned data is abnormal, and then the corresponding data is supplemented, and finally the existing association construction model is used to analyze the correlation.

[0152] This embodiment solves the problem of incorrect data cleaning in the training method of the traditional multimodal association construction model. Specifically, this embodiment calculates the true value of the abnormal data by adding an abnormal data acquisition module and a control module, reduces the amount of deleted data, and ensures the quality of data cleaning. Since the control module refers to the student's historical grades when calculating, the calculated value of the abnormal data is estimated as much as possible, and the student's attendance can be judged based on the student's leave reference index this semester, it can reflect whether the student has missed a large number of knowledge points.

[0153] The contents disclosed above are only preferred feasible embodiments of the present invention, and do not limit the protection scope of the present invention. Therefore, all equivalent technical changes made using the contents of the present invention description and drawings are included in the protection scope of the present invention. In addition, the elements therein can be updated as technology develops.

Claims

1. A multimodal data processing method, characterized in that: The following steps are involved: S1: The data collection module receives and processes relevant data and obtains information on math scores and physics scores, and transmits it to the control module; S2: The control module calculates the sorted math scores according to the math scores, calculates the sorted physics scores according to the physics scores, calculates the correlation factor between the math scores and the physics scores according to the covariance of the sorted math scores and the sorted physics scores, the standard deviation of the sorted math scores, and the standard deviation of the sorted physics scores, and obtains the correlation information between the math scores and the physics scores according to the correlation factor between the math scores and the physics scores and transmits it to the communication module; S3: The communication module transmits the correlation information between the mathematics scores and the physics scores to the user end.

2. A multimodal data processing method according to claim 1, characterized in that: The data collection module includes a multimodal data input submodule, a data conversion submodule and a data integration submodule, wherein the data conversion submodule and the data integration submodule are both communicatively connected to the multimodal data input submodule, and the data integration submodule is communicatively connected to the control module; The multimodal data input submodule is used to receive relevant data of image, text and voice input, transmit relevant data of image and voice input to the data conversion submodule, and transmit relevant data of text input to the data integration submodule; The data conversion submodule is used to convert the relevant data of the image and voice input, and transmit the converted relevant data to the data integration submodule; The data integration submodule integrates the text input related data and the converted related data, obtains the information of the mathematics score and the physics score, and transmits it to the control module.

3. A multimodal data processing method as claimed in claim 2, characterized in that: When the control module calculates the correlation factor between the math scores and the physics scores, the following formula is satisfied: X=[x n ,x n+1 ,…,x N ]; And=[and n ,and n+1 ,…,and N ]; ΔX=[x n *a, x n+1 *a,…,x N *a]; ΔY=[y n *a,y n+1 *a,…,y N *a]; Among them, GLX is the correlation factor between mathematics scores and physics scores, XFC(ΔX, ΔY) is the covariance of the adjusted mathematics scores and the adjusted physics scores, BZC Δx is the standard deviation of the math scores after adjustment, BZC ΔY is the standard deviation of the adjusted physics scores, ΔX is the adjusted mathematics scores, and ΔY is the adjusted physics scores; X is the math score, x n is the math score of the nth student, x N is the math score of the Nth student; Y is the physics score, y n is the physics score of the nth student, y N is the physics score of the Nth student; α is a conversion index, and α has the following values: α=1 or α=0.98 or α=0.

95. When α=1, the student's score is input via text; when α=0.98, the student's score is input via voice; and when α=0.95, the student's score is input via image.

4. A multimodal data processing method as claimed in claim 3, characterized in that: When the control module calculates the correlation information between the math scores and the physics scores, the following formula is satisfied: Among them, β is the correlation information between mathematics scores and physics scores, γ1 to γ4 are the selection thresholds of different correlation factors between mathematics scores and physics scores, when β = 1, it indicates that there is a negative linear relationship between mathematics scores and physics scores, when β = 2, it indicates that there is no linear relationship between mathematics scores and physics scores, and when β = 3, it indicates that there is a positive linear relationship between mathematics scores and physics scores.

5. A multimodal data processing method as claimed in claim 4, characterized in that: In step S1, the following steps are also included: The data collection module transmits the information of mathematics scores and physics scores to the data cleaning module; The data cleaning module deletes abnormal data and missing data from the math scores, and obtains information on the total number of math scores before data cleaning, the math score of the z-th student after data cleaning, the total number of math scores after data cleaning, the total duration of data cleaning, and the math score of the Z-th student after data cleaning, and transmits the information to the control module; In step S2, the following steps are also included: The control module calculates the math score after data cleaning according to the math score of the Zth student after data cleaning and the math score of the zth student after data cleaning, calculates the data cleaning determination factor according to the total number of math scores before data cleaning, the math score of the nth student, the total number of math scores after data cleaning, and the math score of the zth student after data cleaning, and obtains alarm information according to the data cleaning determination factor and transmits it to the communication module; In step S3, the following steps are also included: The communication module transmits the alarm information to the user end and the alarm end.

6. A multimodal data processing method as claimed in claim 5, characterized in that: The data cleaning module includes a data cleaning submodule and a data analysis submodule that are communicatively connected, and the data analysis submodule is communicatively connected to the control module; The data cleaning submodule cleans the abnormal data and missing data of the mathematics scores and obtains the relevant data after the data cleaning, and transmits it to the data analysis submodule; The data analysis submodule is used to analyze and obtain information on the total number of students with the best math scores before data cleaning, the math scores of the zth student after data cleaning, the total number of students with the best math scores after data cleaning, the total duration of data cleaning, and the math scores of the Zth student after data cleaning, and transmit it to the control module.

7. A multimodal data processing method according to claim 6, characterized in that: The communication module includes an information transmission submodule and an alarm submodule that are communicatively connected, and the information transmission submodule and the alarm submodule are both communicatively connected to the control module; The control module transmits the correlation information between the math scores and the physics scores to the information transmission submodule, and synchronously transmits the alarm information to the information transmission submodule and the alarm submodule; The information transmission submodule transmits the correlation information of the mathematics scores and the physics scores and the alarm information to the user end; The alarm submodule transmits the alarm information to the alarm terminal and implements the alarm according to the alarm information.

8. A multimodal data processing method as claimed in claim 7, characterized in that: When the control module calculates the data cleaning determination factor, the following formula is satisfied: X c =[x z ,x z+1 ,…,x z ]; Among them, QX is the data cleaning determination factor, N is the total number of math scores before data cleaning, and x z is the math score of the zth student after data cleaning, Z is the total number of students with math scores after data cleaning, and T is the total time of data cleaning; X c is the math score after data cleaning, x z This is the math score of the Zth student after data cleaning.

9. A multimodal data processing method as claimed in claim 8, characterized in that: When the control module calculates the alarm information, the following formula is satisfied: Among them, BJ is the alarm information, qx ref It is the selection threshold of the data cleaning judgment factor. When BJ=1, the data cleaning is normal and no alarm is needed. When BJ=2, the data cleaning is abnormal and an alarm is needed.

Images