A multimodal post-exam violation judgment method in the open education system

Abstract

The embodiment of the present invention discloses a method for determining post-exam violation behavior based on multimodality under an open education system, including: obtaining a training data set about post-exam behavior, and using the training data set to train a post-exam violation behavior determination model; using the trained model to predict new single images and / or image-text pairs to obtain post-exam violation behavior categories; wherein, during the training process, a single image sample is input into a Resnet network to obtain a first classification feature of the single image sample; an image-text pair sample is input into a CLIP network to obtain a second classification feature of the image-text pair sample; the first classification feature is fused with the second classification feature, and the fused feature is input into an MLP fully connected layer to obtain a third classification feature; the third classification feature is passed through an activation function to obtain the post-exam violation behavior category of the input data. This embodiment automatically identifies post-exam violation behavior through multimodal fusion.

Description

Technical Field

[0001] The embodiments of the present invention relate to the fields of artificial intelligence and educational technology, and in particular to a multimodal post-exam violation determination method in an open education system. Background Art

[0002] The examination system is a large-scale examination release and organization platform, involving the release of examinations, scheduling of invigilators, scheduling of examinees, statistical release of examination scores, and post-exam status verification. Most of these functions require manual operation, which places high demands on the business staff's business familiarity and working hours. The post-exam status verification is based on the examination materials uploaded by the invigilator, including images or images and corresponding descriptions of examinees to determine the examination behavior.

[0003] At present, the marking of post-exam verification status is manually classified and marked based on the descriptive information or pictures uploaded by the invigilator of the learning center. Due to the large number of test data, manual processing is more labor-intensive and has a greater impact from subjective factors.

[0004] In addition, there are some technical means for monitoring examination rooms based on real-time data collection from video streams. For example, patent CN114187661A discloses a method for detecting examinee violations, and patent CN112417990A discloses a method and system for identifying student violations in examinations. However, such methods rely only on one data source, the video stream, with a single information modality. Moreover, the collection and storage of video data requires a large amount of space, which is not conducive to large-scale use. Summary of the invention

[0005] The embodiment of the present invention provides a multimodal post-exam violation determination method in an open education system to solve the above technical problems.

[0006] In a first aspect, an embodiment of the present invention provides a method for determining post-exam violation behavior based on multimodality in an open education system, comprising:

[0007] Acquire a training data set about post-test behavior, and use the training data set to train a post-test violation behavior determination model, wherein the training data set includes single image samples and image-text pair samples;

[0008] Using the trained model to predict new single images and / or image-text pairs, and obtaining the post-test violation category of the new single images and / or image-text pairs;

[0009] Among them, the following operations are performed inside the model during the training process:

[0010] Inputting a single image sample into a Resnet network to obtain a first classification feature of the single image sample;

[0011] Inputting the image-text pair sample into the CLIP network to obtain a second classification feature of the image-text pair sample;

[0012] The first classification feature is fused with the second classification feature, and the fused feature is input into the MLP fully connected layer to obtain the third classification feature;

[0013] The third classification feature is passed through an activation function to obtain the post-test violation category of the input data.

[0014] In a second aspect, an embodiment of the present invention provides an electronic device, the electronic device comprising:

[0015] one or more processors;

[0016] a memory for storing one or more programs,

[0017] When the one or more programs are executed by the one or more processors, the one or more processors implement the multimodal post-exam violation determination method under the open education system described in any embodiment.

[0018] In a third aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multimodal post-exam violation determination method under the open education system described in any embodiment.

[0019] The embodiment of the present invention provides a multimodal post-test violation judgment method under an open education system, which utilizes the advantages of artificial intelligence technology in processing multimodal data. Based on the situation that the uploaded materials contain images and image-text pairs, the network design adopts the fusion method of Resnet and CLIP networks to analyze and process the images and image-text pairs uploaded in the test respectively. The parameters and models obtained through training can be used to classify and mark future post-test verification data, improve the accuracy of classification, and solve the problems of large workload of manual marking and large influence of subjective factors. At the same time, the collection and storage requirements are small, which is conducive to large-scale use. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1It is a flow chart of a method for determining post-exam violation behavior based on multimodality in an open education system provided by an embodiment of the present invention;

[0022] Figure 2 It is a training network structure diagram of a post-test violation behavior determination model provided by an embodiment of the present invention;

[0023] Figure 3 This is an external service architecture diagram of a multi-modal post-exam violation determination method in an open education system provided by an embodiment of the present invention;

[0024] Figure 4 It is a training network structure diagram of another post-test violation behavior determination model provided by an embodiment of the present invention;

[0025] Figure 5 It is a training network structure diagram of another post-test violation behavior determination model provided by an embodiment of the present invention;

[0026] Figure 6 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present invention.

[0028] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0029] In the description of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0030] Before introducing the present application scheme, the terms involved in the scheme are first explained.

[0031] Multimodal: information of multiple different data types, including text, images, audio, video, sensor data, etc.

[0032] Loss function: A function used in machine learning and deep learning models to measure the difference between the model's predicted results and the actual results.

[0033] Activation function: A key component in a neural network used to introduce nonlinearity, enabling the network to learn and represent complex patterns.

[0034] Based on the above explanation, Figure 1 1 is a flowchart of a method for determining post-exam violation behavior based on multimodality in an open education system provided by an embodiment of the present invention. The method is applicable to the case where the uploaded material contains images and image-text pairs, and is executed by an electronic device. Figure 1 As shown, the method specifically includes:

[0035] S110, obtaining a training data set about post-test behavior, and using the training data set to train a post-test violation behavior determination model, wherein the training data set includes single image samples and image-text pair samples.

[0036] S120: Use the trained model to predict a new single image and / or image-text pair to obtain a post-test violation category of the new single image and / or image-text pair.

[0037] Among them, the following operations are performed inside the model during the training process:

[0038] Step 1: Input a single image sample into the Resnet network to obtain the classification features of the single image sample. Optionally, the following operations are performed inside the Resnet network: first, input a single image sample into the Resnet encoder to obtain a single image encoding feature; then, the single image encoding feature is padded with a zero vector so that the padded feature is consistent with the vector dimension of the image-text pair described below; finally, the padded feature is reduced in dimension through a fully connected layer to obtain the classification features of the single image sample.

[0039] At the same time, the image-text pair sample is input into the CLIP network to obtain the classification features of the image-text pair sample. Optionally, the following operations are performed inside the CLIP network: first, the image and text in the image-text pair sample are respectively input into an image encoder and a text encoder to obtain two encoding features respectively. For the convenience of distinction and description, the encoding feature output by the image encoder can be referred to as the first encoding feature, and the feature output by the text encoder can be referred to as the second encoding feature; then, the first encoding feature and the second encoding feature are respectively mapped to the multimodal embedding space and converted into feature vectors of the same dimension; then, the similarity is calculated based on the feature vectors of the same dimension to determine whether the image-text pair sample matches; finally, the feature vector of the matched image-text pair sample is input into another fully connected layer for dimensionality reduction to obtain the classification features of the image-text pair sample.

[0040] For the convenience of distinction and description, in this embodiment, the classification features of single image samples are called first classification features, and the classification features of image-text pair samples are called second classification features; the one fully connected layer in the Resnet network is called the first fully connected layer, and the other fully connected layer in the CLIP network is called the second fully connected layer.

[0041] Step 2: Fuse the first classification feature with the second classification feature, and input the fused feature into the MLP fully connected layer to obtain another classification feature, called the third classification feature.

[0042] Step 3: The third classification feature is subjected to an activation function to obtain a post-test violation behavior category of the input data. Optionally, the post-test violation behavior category includes at least one of: absence from the test, violation of discipline, cheating, disrupting the test, and normal.

[0043] The method of the present application is further illustrated by means of several embodiments below.

[0044] Example 1

[0045] In this embodiment, the Figure 2 The training network structure shown in the figure has the following specific structures and functions of each part:

[0046] 1. Input layer: The input layer data is divided into single images and image-text pairs.

[0047] 2. Network architecture:

[0048] 1) For single image data:

[0049] (1) Extract pure image features through Resnet50.

[0050] (2) Zero vector concatenation: Fill the zero vector so that the feature vector dimension of a single image is consistent with the vector dimension of the image-text pair. Specifically, since a single image lacks a text vector, the dimension differs from that of the image-text pair by the dimension of the text vector. For example, the feature output of a single image is (batch, 1024), and the feature output of the image-text pair is (batch, 1280). The difference between the two is the feature dimension of the text (batch, 256). In order to keep the vector dimension consistent, a (batch, 256) empty vector is concatenated to the single image output feature to prepare for subsequent fusion.

[0051] (3) Fully connected layer: The high-dimensional features generated after the image passes through Resnet are input into the fully connected layer to convert the high-dimensional features into low-dimensional features required for the classification task.

[0052] 2) For image-text pairs:

[0053] The image-text pair is processed through the CLIP network layer, which includes:

[0054] (1) Encoding layer: including the image encoder and text encoder in the figure, which is responsible for converting input data such as images and text into structured matrix vectors.

[0055] (2) Multimodal embedding space: Convert data of different modalities into vectors of the same dimension and map them to the same vector space for representation, which facilitates subsequent similarity calculations, etc.

[0056] (3) Similarity calculation: The process of calculating the similarity between image and text vectors. The higher the similarity, the higher the semantic relevance. The similarity is used to determine whether the image and text match, thereby finding the image and its corresponding text.

[0057] (4) Fully connected layer: The results of similarity calculation are mapped to the category space through the fully connected layer and converted into specific classification results.

[0058] 3) Fusion: The image modality features output by the Resnet network and the multimodal features output by the CLIP network are directly spliced ​​together in the feature dimension to form a larger feature dimension. Fusion can be the fusion of any features between different modal data.

[0059] 3. MLP fully connected layer: learns the relationship between different modalities, integrates the fused information into a unified representation, extracts classification features, and converts the fused features into corresponding classification features.

[0060] 4. Softmax: Activation function that converts the output of the fully connected layer into probability.

[0061] Furthermore, the above network structure can be implemented based on Pytorch during the training phase, and the specific implementation steps may include:

[0062] 1. Model training

[0063] The network design is as follows:

[0064] Single image data uses the Resnet50 basic network for image processing.

[0065] For image-text data, image Encoder uses Resnet50 as the basic model network; textEncoder uses the Bert model as the basic network model.

[0066] The similarity is calculated using cosine similarity.

[0067] When fusion of modalities, the dim=1 parameter is used to fuse the image tensor and the text tensor together, and the RELU activation function is used during back propagation.

[0068] The loss function uses the cross entropy loss function CrossEntropyLoss, the optimizer uses the Adam optimizer, and the learning rate uses lr=1e-4.

[0069] The activation function of the output layer uses Softmax to calculate the probability of each output.

[0070] 2. Model Deployment

[0071] After training, the trained model is deployed in a docker container.

[0072] 1) Optimize the model:

[0073] Prune and quantize the trained model to reduce unnecessary parameters such as weights or offsets, thereby reducing the size of the model; reduce the type precision of the model data and increase the inference speed.

[0074] 2) Model export:

[0075] Implement the network based on Pytorch and export the model to pt or pth format.

[0076] 3) Model deployment:

[0077] This embodiment is based on docker container deployment, and the model parameters are packaged into a docker container, which is convenient for one-time packaging and multiple deployments.

[0078] 3. Application

[0079] 1) External access:

[0080] The application system is implemented based on the FastAPI framework of Python, exposing external access interfaces to facilitate web or client program development and calls. Figure 3 shown.

[0081] Among them, the client input is the collected test data: the images during the test and the text after the test are input into the system through the client, and the trained model processes and returns the judgment results and outputs them to the client.

[0082] In this embodiment, the advantages of artificial intelligence technology in processing multimodal data are utilized. Based on the situation that the uploaded materials contain images and image-text pairs, the network design adopts the fusion method of Resnet50 and CLIP network to analyze and process the images and image-text pairs uploaded in the test. The parameters and models obtained through training can be used to classify and mark the future post-test verification data, thus solving the problem of large workload of manual marking and large influence of subjective factors.

[0083] In addition, the late fusion method is used in this embodiment. Compared with the early fusion method, it can fully utilize the advantages of the pre-trained model, greatly reduce the amount of calculation, and reduce the interference between different modalities. The model trained based on the above method can be deployed in the production environment to achieve automated verification in the post-exam verification process in the future, reduce manual participation, and improve objectivity.

[0084] Example 2

[0085] Usually in natural language processing, since short text (such as just one or a few words) contains limited information, the information error of the encoded features obtained after encoding is increased compared with the encoded features of long text. Considering that the text data uploaded in this embodiment also varies in length, this embodiment can give priority to using image-text pairs including long text for training in model training, and then take into account the prediction of short text samples through text expansion and feature conversion. The specific training process is as follows:

[0086] First, the model is trained using image-text pair samples including long texts. After the training is completed, the trained model is used to test image-text pair samples including short texts. For the convenience of distinction and description, the image-text pair samples including long texts can be referred to as first image-text pair samples, and the image-text pair samples including short texts can be referred to as second image-text pair samples.

[0087] If it is found in the test that the similarity calculation performance of each second image-text pair sample is not up to standard, the text in each second image-text pair sample can be expanded, and the similarity can be recalculated using the expanded sample. For example, when the second image-text pair samples are processed using the trained model, it is found that the matching image-text pairs cannot be accurately identified in the similarity calculation, and the recognition accuracy is less than the set standard. This may be due to the single information in the short text, which cannot provide good text encoding features. In this case, each short text can be expanded to increase the amount of text information and the encoding effect. Optionally, the expansion can be done manually, or it can be done by automated methods such as machine translation or large models, which is not specifically limited in this embodiment.

[0088] Select multiple expanded texts that meet the similarity calculation performance after expansion, and record the second encoding features of each expanded text and the second encoding features of each original text, that is, record the features of the expanded text and the original text after being encoded by the text encoder. Due to the diversity of expansion results, this embodiment selects multiple expanded texts that can improve the similarity calculation performance and their corresponding original texts to learn their expansion rules.

[0089] In a specific implementation, when learning the expansion rule, another text encoder that is parallel to the text encoder and has the same parameters can be added to the CLIP network, and a feature converter can be added after the other text encoder, such as Figure 4 As shown. For the convenience of distinction and description, the text encoder originally existing in the CLIP network is called the first text encoder, and the other text encoder added here is called the second text encoder. The second text encoder loads the parameters trained in the first text encoder, and a trainable feature converter is connected between the second text encoder and the multimodal embedding space. The feature converter can use MLP (the MLP can include linear layers or nonlinear layers), or other feature conversion structures can be used, which is not specifically limited in this embodiment.

[0090] Then, the feature converter is trained using the second coding features of each selected expanded text and the second coding features of the corresponding original text, so that the second coding features of the original text are continuously approached to the second coding features of the corresponding expanded text after being trained by the feature converter. By training the feature converter, the law of the second coding features between the original text and the expanded text can be learned, thereby characterizing the expansion law between the original text and the expanded text.

[0091] After the feature converter is trained, in the prediction of a new image-text pair, it can be determined whether to use the first text encoder or to use the second text encoder and the feature converter to calculate the similarity according to the length of the text in the new image-text pair. Specifically, after the training is completed, two parallel branches are formed in the text encoding link, one of which includes the first text encoder and the other includes the second text encoder and the feature converter. If the text included in the new image-text pair is a long text (for example, the text capacity is greater than the set length), it is input into the first branch for encoding, and subsequent operations such as multimodal embedding space conversion and similarity calculation are performed based on the encoded features; if the text included in the new image-text pair is a short text, it is input into the second branch for encoding, and subsequent operations such as multimodal embedding space conversion and similarity calculation are performed based on the encoded features.

[0092] Furthermore, during the training process of the above-mentioned feature converter, since the styles of the original texts are different, the expansion results are also different. Using a unified feature converter to learn all the expansion rules may have the problem of difficult convergence or too long convergence time. At this time, the original texts corresponding to the selected expanded texts can be clustered in advance, and the second encoded features of these texts can be clustered into multiple cluster clusters; then, in the link of adding a second text encoder, multiple second text encoders that are parallel to the first text encoder and have the same parameters are added, and feature converters are added after each second text encoder, wherein the multiple cluster clusters correspond one-to-one to the multiple second text encoders and the multiple feature converters, such as Figure 5 shown.

[0093] Accordingly, the feature converters are trained using the expanded texts and original texts in each cluster. Specifically, the following operations are performed for each cluster: the feature converter corresponding to the current cluster is trained using the second coding features of the expanded texts and the second coding features of the original texts in the current cluster, so that the second coding features of the original texts in the current cluster continuously approach the second coding features of the corresponding expanded texts after being trained by the feature converter.

[0094] After the training is completed, assuming that there are N clusters, N+1 parallel branches are formed in the text encoding link, wherein the first branch only includes the first text encoder, and the remaining N branches each include a second text encoder and a feature converter. In the prediction of a new image-text pair, if the new image-text pair includes a long text, the first branch is used to encode the long text, and subsequent operations such as multimodal embedding space conversion and similarity calculation are performed based on the encoded features; if the new image-text pair includes a short text, the distance from the short text to the cluster center of each cluster is first calculated, and the branch corresponding to the cluster center closest to the short text is used to encode the short text, and subsequent operations such as multimodal embedding space conversion and similarity calculation are performed based on the encoded features.

[0095] In this embodiment, in order to solve the problem of insufficient information in short texts, samples including long texts are preferentially selected for model training. After training, the short text samples are tested, and the similarity calculation performance is improved by text expansion. Then, the expansion rules between the expanded text and the original text are learned by adding feature converters, so that the directional expansion of the short text is completed through feature conversion in the new short text encoding. In order to take into account the diversity of texts and expansion rules, the original short texts can also be clustered and the corresponding feature converters can be classified and trained to further improve the convergence speed of training and the accuracy of expansion features.

[0096] Figure 6 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention is shown in FIG. Figure 6 As shown, the device includes a processor 60, a memory 61, an input device 62 and an output device 63; the number of processors 60 in the device can be one or more. Figure 6 A processor 60 is taken as an example; the processor 60, the memory 61, the input device 62 and the output device 63 in the device can be connected by a bus or other means. Figure 6 The example of connecting through bus is taken in the following.

[0097] The memory 61, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the multimodal post-exam violation determination method under the open education system in the embodiment of the present invention. The processor 60 executes various functional applications and data processing of the device by running the software programs, instructions and modules stored in the memory 61, that is, realizing the above-mentioned multimodal post-exam violation determination method under the open education system.

[0098] The memory 61 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 61 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 61 may further include a memory remotely arranged relative to the processor 60, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0099] The input device 62 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the device. The output device 63 may include a display device such as a display screen.

[0100] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the multimodal post-exam violation determination method in an open education system of any embodiment is implemented.

[0101] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0102] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0103] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0104] Computer program code for performing the operation of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as C or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0105] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal post-exam violation judgment method in an open education system, characterized in that: include: Acquire a training data set about post-test behavior, and use the training data set to train a post-test violation behavior determination model, wherein the training data set includes single image samples and image-text pair samples; Using the trained model to predict new single images and / or image-text pairs, and obtaining the post-test violation category of the new single images and / or image-text pairs; Among them, the following operations are performed inside the model during the training process: Inputting a single image sample into a Resnet network to obtain a first classification feature of the single image sample; Inputting the image-text pair sample into the CLIP network to obtain a second classification feature of the image-text pair sample; The first classification feature is fused with the second classification feature, and the fused feature is input into the MLP fully connected layer to obtain the third classification feature; The third classification feature is subjected to an activation function to obtain the post-test violation category of the input data; The step of inputting the image-text pair sample into the CLIP network to obtain the second classification feature of the image-text pair sample includes: Inputting the image and text in the image-text pair sample into the image encoder and the first text encoder respectively, and obtaining the first encoding feature and the second encoding feature respectively; Mapping the first encoding feature and the second encoding feature to the multimodal embedding space respectively and converting them into feature vectors of the same dimension; Calculate similarity based on feature vectors with the same dimensions to determine whether the image-text pair samples match; Inputting the feature vector of the matched image-text pair sample into the second fully connected layer for dimensionality reduction to obtain the second classification feature of the image-text pair sample; After the model is trained using the first image-text pair sample including the long text, the trained model is used to test the second image-text pair sample including the short text; If the similarity calculation performance of each second image-text pair sample does not meet the standard, the text in each second image-text pair sample is expanded, and the similarity is recalculated using the expanded sample; Select multiple expanded texts whose similarity calculation performance meets the standard after expansion, and record the second coding feature of each expanded text and the second coding feature of each original text; Adding a second text encoder in parallel with the first text encoder and with the same parameters to the CLIP network, and adding a feature converter after the second text encoder; The feature converter is trained using the second coding features of each expanded text and the second coding features of each original text, so that the second coding features of the original text continuously approach the second coding features of the corresponding expanded text after being trained by the feature converter; In the prediction of a new image-text pair, it is determined whether to use the first text encoder or to use the second text encoder and a feature converter to calculate similarity according to the text length in the new image-text pair.

2. The method according to claim 1, characterized in that The step of inputting a single image sample into a Resnet network to obtain a first classification feature of the single image sample includes: Input the single image sample into the Resnet encoder to obtain the single image encoding feature; Performing zero vector padding on the single image encoding feature so that the padded feature is consistent with the vector dimension of the image-text pair; The filled features are reduced in dimension by the first fully connected layer to obtain the first classification features of the single image sample.

3. The method according to claim 1, characterized in that The adding of a second text encoder in parallel with the first text encoder and having the same parameters in the CLIP network, and adding a feature converter after the second text encoder, comprises: clustering the second encoding features of each original text to obtain a plurality of clustering clusters; adding a plurality of second text encoders in parallel with the first text encoder and having the same parameters in the CLIP network, and adding a feature converter after each second text encoder, wherein the plurality of clustering clusters correspond one-to-one to the plurality of second text encoders and the plurality of feature converters; The feature converter is trained by using the second coding features of each expanded text and the second coding features of each original text, so that the second coding features of the original text are continuously close to the second coding features of the corresponding expanded text after being trained by the feature converter, including: for each cluster, using the second coding features of each expanded text in the current cluster and the second coding features of each original text, training the feature converter corresponding to the current cluster, so that the second coding features of each original text in the current cluster are continuously close to the second coding features of the corresponding expanded text after being trained by the feature converter; The method comprises: in the prediction of the new image-text pair, if the new image-text pair includes a long text, using the first text encoder to encode the long text and calculating the similarity; if the new image-text pair includes a short text, calculating the distance from the short text to the cluster center of each cluster, using the second text encoder and feature converter corresponding to the cluster center closest to the short text to encode the short text and calculating the similarity.

4. The method according to claim 1, characterized in that: The image encoder uses Resnet50 as the basic network model, and the first text encoder uses the Bert model as the basic network model.

5. The method according to claim 1, characterized in that The fusing the first classification feature with the second classification feature includes: Use the dim=1 parameter to fuse the first classification feature with the second classification feature.

6. The method according to claim 1, characterized in that The post-exam violation behavior categories include: absence from the exam, violation of discipline, cheating, disrupting the exam, and at least one of normal.

7. An electronic device, characterized in that: include: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the multimodal post-exam violation determination method under the open education system described in any one of claims 1-6.

8. A computer-readable storage medium, characterized in that: A computer program is stored thereon, which, when executed by a processor, implements the multimodal post-examination violation determination method in an open education system as described in any one of claims 1-6.

Images