A method, device, equipment and medium for matching a question stem image and an image

Abstract

The present disclosure provides a matching method and device for answer images and stem images, equipment and medium, relating to the technical field of image detection and analysis. The method comprises: extracting a plurality of reference features of an answer image contained in a test question image; performing iterative feature fusion on the answer image using the plurality of reference features to obtain a plurality of answer sub-images; and performing feature matching on a plurality of stem images contained in the test question image and the plurality of answer sub-images to obtain the answer sub-image corresponding to each stem image. The matching method for answer images and stem images using the exemplary embodiments of the present disclosure matches answer images and stem images, and the matching result is more accurate.

Description

Technical Field

[0001] This invention relates to the field of image detection and analysis technology, and in particular to a method, apparatus, device, and medium for matching answer images and question stem images. Background Technology

[0002] In recent years, with the development of deep learning technology, image segmentation methods based on deep learning have made great progress. Currently, there are related technical solutions for analyzing and segmenting textbook images; however, these solutions struggle to achieve accurate segmentation when faced with various complex situations. Summary of the Invention

[0003] According to one aspect of this disclosure, a method for matching a response image and a question stem image is provided, the method comprising:

[0004] Extract multiple reference features from the answer image contained in the question image;

[0005] Iterative feature fusion is performed on the response image using multiple reference features to obtain multiple response sub-images;

[0006] Feature matching is performed on the multiple question stem images and multiple answer sub-images contained in the question image to obtain the answer sub-image corresponding to each question stem image.

[0007] According to another aspect of this disclosure, a matching device for answer images and question stem images is provided, the device comprising:

[0008] The extraction module is used to extract multiple reference features of the answer image contained in the question image;

[0009] The module is used to perform iterative feature fusion on the answer image using multiple reference features to obtain multiple answer sub-images, and to perform feature matching on multiple question stem images and multiple answer sub-images contained in the question image to obtain the answer sub-image corresponding to each question stem image.

[0010] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0011] Processor; and,

[0012] Memory for stored programs;

[0013] The program includes instructions that, when executed by the processor, cause the processor to perform the method according to an exemplary embodiment of the present disclosure.

[0014] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions for causing the computer to perform the method described according to exemplary embodiments of this disclosure.

[0015] One or more technical solutions provided in this disclosure extract multiple reference features from the answer images contained in the question image, and use these multiple reference features to perform iterative feature fusion on the answer images to obtain multiple answer sub-images. Finally, feature matching is performed on the multiple question stem images and multiple answer sub-images contained in the question image to obtain the answer sub-image corresponding to each question stem image, so as to facilitate grading and evaluation and improve the efficiency and accuracy of grading. When the iterative feature fusion method is used to segment the answer image into multiple answer sub-images, in each iteration, the incorrectly segmented areas are further segmented and merged until all reference features are updated, eliminating segmentation errors and making the final segmentation results of multiple answer sub-images more accurate and unaffected by complex layouts. Attached Figure Description

[0016] Further details, features, and advantages of this disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0017] Figure 1 A schematic diagram of an example system in which the various methods described herein may be implemented according to exemplary embodiments of the present disclosure;

[0018] Figure 2 A schematic flowchart illustrating a method for matching answer images and question stem images according to exemplary embodiments of the present disclosure is shown;

[0019] Figure 3 A schematic flowchart illustrating a method for obtaining answer images and question stem images according to exemplary embodiments of the present disclosure is shown;

[0020] Figure 4 A schematic flowchart illustrating a method for determining a response sub-image according to an exemplary embodiment of the present disclosure is shown;

[0021] Figure 5 A schematic block diagram of the functional modules of a matching device for answer images and question stem images according to an exemplary embodiment of the present disclosure is shown;

[0022] Figure 6 A schematic block diagram of a chip according to an exemplary embodiment of the present disclosure is shown;

[0023] Figure 7 A structural block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0024] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0025] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0026] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc., used in this disclosure are only used to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0027] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0028] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0029] Before introducing the embodiments of this disclosure, the relevant terms involved in the embodiments of this disclosure are first defined as follows:

[0030] Feature fusion is an important method in the field of pattern recognition. Image recognition in computer vision, as a special pattern classification problem, still faces many challenges. Feature fusion can comprehensively utilize multiple image features, achieve complementary advantages of multiple features, and obtain more robust and accurate recognition results.

[0031] Connected component analysis: Identifying and marking connected regions in an image; also known as connected component labeling.

[0032] Optical Character Recognition (OCR) refers to the process of analyzing and recognizing textual data in image files to obtain text and layout information. In other words, it involves recognizing the text in an image and returning it as text.

[0033] Convolutional Recurrent Neural Networks (CRNNs) are primarily used for end-to-end recognition of text sequences of variable length. Instead of segmenting individual characters first, they transform text recognition into a time-dependent sequence learning problem, which is image-based sequence recognition.

[0034] Convolutional Neural Networks (CNNs) are a type of feedforward neural network that includes convolutional computations and has a deep structure. They are one of the representative algorithms of deep learning.

[0035] Feed-forward denoising convolutional neural networks (DnCNN) are modified versions of VGG. The network structure is a cascaded structure of (convolution, BN, ReLU). Unlike ResNet, there are no jump connections inside the model. Instead, residual learning is used at the network output.

[0036] Semantic segmentation is a crucial area in computer vision, referring to the classification of pixels in an image based on their categories, achieving pixel-level image classification. It can divide an image or video into multiple blocks according to the similarities and differences in their categories.

[0037] Mask R-CNN is an instance segmentation model that can determine the location and category of each object in an image, providing pixel-level predictions. "Instance segmentation" refers to segmenting each object of interest within a scene, regardless of whether they belong to the same category—for example, the model can identify individual objects such as vehicles and people in street view videos.

[0038] SIFT (Scale-invariant feature transform) is an algorithm for detecting local features. This algorithm obtains features by finding the feature points (interest points or corner points) in an image and their related scale and orientation descriptors, and then performs image feature point matching, achieving good results.

[0039] HOG Feature Extraction Algorithm: HOG (Histogram of Oriented Gradient), also known as the Histogram of Oriented Gradients, is a feature descriptor similar to the Canny edge detector. It is used for object detection in computer vision and image processing.

[0040] In recent years, with the development of deep learning technology, image segmentation methods based on deep learning have made great progress. These methods utilize deep neural networks to learn the relationship between image features and pixels, and have wide applications in many fields. In particular, the application of convolutional neural networks and semantic segmentation models has provided new ideas and methods for solving problems such as layout analysis of textbook images and segmentation of question answers.

[0041] However, in practical applications, due to the complex and diverse layout of teaching materials, including different fonts, sizes, colors, and typesets, as well as factors such as students' non-standard answers (e.g., writing answers to multiple different questions in one answer area or writing answers in other areas after correction), current technology still faces challenges in accurately segmenting answer content by question.

[0042] To address the aforementioned problems, this exemplary embodiment provides a method for matching answer images and question stem images. When a question image contains both question stem information and answer information, multiple question stem images and multiple answer images can be obtained based on the question image. By extracting multiple reference features from the multiple answer images contained in the question image, and using these multiple reference features to perform iterative feature fusion on the answer images, multiple segmented answer sub-images are obtained. Then, by performing feature matching on the multiple question stem images and multiple answer sub-images, the answer sub-image corresponding to each question stem image can be obtained, so that graders can grade and evaluate more effectively, thereby improving the accuracy of question analysis. It should be understood that, in terms of the source of the questions, the questions here can be questions from different subjects, different versions of questions, questions from different publishers, etc., and are not limited to these; in terms of example type, the questions here can be multiple-choice questions, application problems, geometry problems, etc., and are not limited to these. Graders can be teachers, students, etc., and are not limited to these.

[0043] Figure 1 A schematic diagram of an example system in which various methods described herein can be implemented according to exemplary embodiments of this disclosure is shown. Figure 1 As shown, the system 100 of the exemplary embodiments of this disclosure may include: a user device 110, a computing device 120, and a data storage system 130.

[0044] like Figure 1As shown, the user equipment 110 can communicate with the computing device 120 via a communication network. This communication network can be a wired communication network or a wireless communication network. The wired communication network can be a communication network based on power line carrier technology, and the wireless communication network can be a local area network (LAN) or a wide area network (WAN). The LAN can be a Wi-Fi network, a Zigbee network, a mobile communication network, or a satellite communication network, etc.

[0045] like Figure 1 As shown, the user equipment 110 may include a computer, mobile phone, or information processing center, etc., as a smart terminal. The user equipment 110 can act as an image acquisition terminal for the test question image and initiate a request to the computing device 120. The computing device 120 may be a cloud server, network server, application server, or management server, etc., with data processing capabilities, to implement the matching method between the answer image and the question stem image of the exemplary embodiment of this disclosure. The server may be configured with a processor, which may include an image feature extraction processor, an image fusion processor, and an image feature matching processor to complete the tasks of extracting multiple reference features of the answer image, image feature fusion, and matching the answer sub-image and the question stem image.

[0046] like Figure 1 As shown, the data storage system 130 described above can store a database of multiple reference features of the response image. The database can be located on the computing device 120 or on another network server. The data storage system 130 can be separate from the computing device 120 or integrated into the computing device 120.

[0047] In practical applications, computer devices can extract multiple reference features from the answer images contained within the test question image, and use these multiple reference features to perform iterative feature fusion on the answer images to obtain multiple answer sub-images. Finally, feature matching is performed on the multiple question stem images and multiple answer sub-images contained within the test question image to obtain the answer sub-image corresponding to each question stem image. Based on this, when a teacher uploads a test question image to a computer device through a user device, the exemplary embodiment of this disclosure can identify the test question image, extract multiple reference features from the multiple answer images contained within the test question image, and use these multiple reference features to perform iterative feature fusion on the answer images to obtain multiple answer sub-images. Finally, matching is achieved between each test question stem image and its corresponding answer sub-image. The matching result is used as the recommendation target and recommended to the teacher user to facilitate the teacher user's review, correction, and information collection of the questions and corresponding student answers.

[0048] The matching method for answer images and question stem images of the exemplary embodiments of this disclosure can be applied to a server or a chip in a server. The matching method for answer images and question stem images of the exemplary embodiments of this disclosure is described in detail below with reference to the accompanying drawings.

[0049] Figure 2 An illustrative flowchart illustrating a method for matching answer images and question stem images according to an exemplary embodiment of this disclosure is provided.

[0050] like Figure 2 As shown, the matching method between the answer image and the question stem image includes:

[0051] Step 201: Extract multiple reference features from the answer image contained in the question image. It should be understood that these multiple reference features can include shallow and deep features of the answer image. For example, shallow features of the answer image can include features such as color, texture, and shape, while deep features of the answer image can include text content features, etc.

[0052] In practical applications, the above-mentioned test question images can be teaching aid images. After the teacher user uploads the teaching aid image to the computing device through the user device, the computing device can first preprocess the teaching aid image, and then extract features from the preprocessed teaching aid image to obtain multiple reference features of the answer image contained in the teaching aid image.

[0053] Image preprocessing in exemplary embodiments of this disclosure may include at least one of the following image preprocessing methods:

[0054] The first method involves binarizing the textbook image. For example, a thresholding method can be used to binarize the textbook image. The grayscale value of each pixel in the textbook image is compared with a threshold. Pixels with values ​​greater than the threshold are considered white, and pixels with values ​​less than the threshold are considered black, so that the entire textbook image presents a clear black and white effect.

[0055] The second method involves noise removal from the textbook images. For example, noise removal can be achieved using filter-based methods, model-based methods, or learning-based methods. Alternatively, the textbook image can be input into a DnCNN neural network model for denoising, resulting in a noisy image. Then, a residual learning approach can be used to obtain the noise-removed textbook image.

[0056] The extraction of multiple reference features of the answer image contained in the test question image in the exemplary embodiments of this disclosure can be performed based on a feature extraction algorithm or based on a deep learning model.

[0057] For example, when using feature extraction algorithms, multiple shallow reference features of the test question image can be extracted using SIFT, HOG, ORB, and HAAR feature extraction algorithms, while multiple deep reference features of the test question image can be extracted using optical character recognition and natural language processing methods.

[0058] When using the HOG feature extraction algorithm to extract multiple shallow reference features from a test question image, the test question image can be read first and converted into a grayscale image. The grayscale image matrix is ​​then normalized to reduce the influence of factors such as lighting and background; gamma correction can be used in this case. Next, a suitable gradient operator is selected to calculate the gradient map, mainly focusing on the gradients in the x and y directions, and the magnitude and direction of the combined gradient are calculated. The detection window is divided into cells of the same size; adjacent cells are combined into larger, overlapping blocks to fully utilize overlapping edge information. The histogram of the entire block is then calculated, and the gradient histogram within each block is normalized. By combining the information from all blocks, the HOG feature descriptor can finally be visualized.

[0059] For example, when using deep learning models for feature extraction, multiple reference features of the test question image can be extracted through Visual Geometry Group (VGG) networks, Deep Residual Network (ResNet) networks, or Convolutional Neural Networks (CNN) networks. Since using deep learning models to extract image features is a conventional technique, it will not be discussed in detail here.

[0060] Step 202: Iteratively fuse the response image using multiple reference features to obtain multiple response sub-images. For example, after fusing features based on one reference feature, the reference feature is updated, and another reference feature is used to continue fusing features on the fused image. After multiple iterations, multiple response character images are obtained. By using iterative feature fusion, in each iteration, incorrectly segmented regions are further segmented and merged until all reference features are updated, eliminating segmentation errors and making the final segmentation results of the multiple response sub-images more accurate and unaffected by complex layouts.

[0061] Step 203: Perform feature matching on the multiple question stem images and multiple answer sub-images contained in the question image to obtain the answer sub-image corresponding to each question stem image. For example, the semantic features of the multiple question stem images and multiple answer sub-images contained in the question image can be extracted and feature matching can be performed to obtain the answer sub-image corresponding to each question stem image.

[0062] For example, an optical character recognition (OCR) module can be used to recognize the text information of multiple question stem images and multiple answer sub-images contained in a test question image. Specifically, multiple question stem images and multiple answer sub-images contained in a test question image can be input into the OCR module for forward reasoning. The OCR module outputs the text information of the corresponding question stem images and the corresponding text information of the answer sub-images. Then, based on feature matching methods, each question stem image is matched with its corresponding answer sub-image. It should be understood that the feature matching method here can be Euclidean space comparison, relaxation comparison, dynamic program comparison, and HMM (Hiddne Markov Mode), etc., and is not limited to these.

[0063] For example, a neural network model can be used to extract semantic information from multiple question stem images and multiple answer sub-images contained in a test question image, and then match the recognition results to obtain the answer sub-image corresponding to each question stem image. It should be understood that the neural network model here can be one of the deep learning models mentioned above, which will not be elaborated further here.

[0064] As one possible implementation method, Figure 3 A schematic flowchart illustrating a method for obtaining the answer image and the question stem image according to an exemplary embodiment of this disclosure is provided. Figure 3 As shown, the method for obtaining the answer image and the question stem image in an exemplary embodiment of this disclosure may include:

[0065] Step 301: Perform layout analysis on the test question image to obtain the location and type information of the multiple sub-region images included in the test question image. For example, a layout analysis model can be used to perform layout analysis on the test question image. The type information of the multiple sub-region images included in the test question image can be title, question stem, options, printed text, handwritten text, page numbers, etc., where printed text corresponds to the question stem and handwritten text corresponds to the student's answer. The location information of the multiple sub-region images included in the test question image can be the position coordinates of each sub-region image.

[0066] When using a layout analysis model to perform layout analysis on test question images, the model must first be trained to obtain the trained model. The test question image then undergoes preprocessing; the preprocessing method can be found in the previous descriptions and will not be repeated here. Next, the preprocessed test question image is input into the trained layout analysis model. This model can then obtain the location and type information of multiple sub-regions within the test question image based on its edges and structure.

[0067] In practical applications, the above-mentioned layout analysis model can be trained by collecting a sample set of layout data of test question images, including test questions from different subjects, versions, and publishers. Then, the sample set of layout data is labeled with different parts such as titles, question stems, options, printed text, handwritten text, page numbers, and other areas. Deep learning technology is then used to train the model so that it can accurately identify each part of the test question image and improve the accuracy of test question image layout analysis.

[0068] For complex test question image layouts, a dataset of complex layout data can be collected, including various layout formats such as multi-column, multi-row, horizontal, and vertical layouts. This dataset is then labeled with different parts, such as titles, question stems, options, printed text, handwritten text, page numbers, and other areas. Finally, deep learning technology is used to train a layout analysis model, enabling it to accurately identify the various parts of the complex layout and improve the accuracy of layout analysis.

[0069] Step 302: If the type information of multiple sub-region images matches the preset type information of the answer image, obtain the answer image from the question image based on the positioning information of the multiple sub-region images. The preset type of the answer image includes the font type of the answer image. It is important to note that before step 302, the layout of the question image can be corrected to ensure that the answer area for each question is in a horizontal or vertical direction, improving the accuracy of subsequent matching. For example, the layout of the question image can be corrected using geometric transformation or channel overlay methods.

[0070] For example, when using geometric transformations to correct the layout of a test question image, the image can be processed through mirror transformation, transpose transformation, scaling transformation, rotation transformation, etc., so that the test question image can be corrected.

[0071] In practical applications, the font type of the above-mentioned answer image can be handwritten or different fonts in printed text. For example, it can be "Songti" or "Kaiti". When the font type of the above-mentioned answer image is handwritten, the sub-region image with handwritten font type information among the multiple sub-region images is identified. If the sub-region image matches the preset type information of the answer image, then the sub-region image with handwritten font type information is considered the answer image.

[0072] After determining the answer image, a neural network model can be used to segment the answer image to improve the accuracy of segmentation of the answer image contained in the question. At this point, a question segmentation model can be used to segment the answer image, obtaining segmented answer images. Then, multiple reference features of the segmented answer images are extracted and iteratively fused to obtain multiple answer sub-images. This question segmentation model can collect a data sample set of answer region data from question images, including question images from different subjects, versions, and publishers. Then, the answer region data sample set is labeled to mark the answer region for each sub-question. Finally, deep learning technology is used for training to enable it to accurately segment the answer region for each sub-question, thus improving the accuracy of answer image segmentation in the question image.

[0073] Step 303: If the image type information of multiple sub-regions matches the preset type information of the question stem image, obtain the question stem image from the question image based on the positioning information of the multiple sub-region images. It should be understood that steps 302 and 303 here can be executed synchronously or asynchronously. The preset type of the question stem image includes the font type of the question stem image.

[0074] In practical applications, the font type of the question stem image can be printed font or a printed font that differs from the font of the answer image. For example, it could be "Songti" or "Kaiti". When the answer image is printed "Songti", the question stem image can be printed "Kaiti". When the font type of the question stem image is printed font, the sub-region image with printed font type information among the multiple sub-region images is identified. If this sub-region image matches the preset type information of the question stem image, then the sub-region image with printed font type information is considered the question stem image.

[0075] As one possible implementation method, Figure 4 A schematic flowchart illustrating a method for determining a response sub-image according to an exemplary embodiment of this disclosure is provided. Figure 4 As shown, the above-mentioned iterative feature fusion of multiple reference features on the response image to obtain multiple response sub-images may include:

[0076] Step 401: Identify multiple target connected regions in the answer image that match the target reference feature. The preset feature values ​​of the reference features corresponding to these multiple target connected regions are different, and the target reference feature is one of these multiple reference features. By identifying multiple target connected regions that match the target reference feature, the answer image can be further segmented and merged according to the preset feature values ​​of different target reference features. This improves the accuracy of answer image segmentation and reduces the possibility that the segmented answer sub-images include answer information from other question stems or only contain a portion of the answer information from the corresponding question stem. It should be understood that the preset feature values ​​here can be set based on experience and are not limited here.

[0077] For example, based on the principle of connected component analysis, regions in the answer image that have the same preset feature value as the target reference feature can be retained, while regions in the answer image that have different preset feature value from the target reference feature can be segmented to obtain multiple target connected regions.

[0078] When the above-mentioned response images are multiple adjacent response images, multiple target connected regions can be obtained by merging regions in the multiple adjacent response images that have the same preset feature value as the target reference feature and segmenting regions in the multiple adjacent response images that have different preset feature values ​​as the target reference feature.

[0079] Step 402: In response to satisfying the iteration continuation condition, update the response image based on multiple target connected regions, and update the target reference features based on multiple reference features. It should be understood that the iteration continuation condition here includes multiple reference features that have not been fully traversed, that is, among the multiple reference features, there are reference features for which the corresponding multiple target connected regions have not yet been determined.

[0080] When the iteration continuation condition is met, it indicates that multiple reference features of the response image have not been fully traversed. Therefore, the target reference feature can be updated based on multiple reference features, and the response image can be updated based on multiple target connected regions, that is, the multiple target connected regions determined by the previous target reference feature are used as the response image.

[0081] For example, multiple reference features can be viewed as a feature set. When the iteration continuation condition is met, any reference feature other than the target reference feature can be extracted from this feature set as the updated target reference feature. When updating the response image based on multiple target connected regions, the updated response image may include the determined multiple target connected regions. Based on this, using multiple target connected regions as the response image and redetermining the multiple target connected regions in the response image that match the target reference feature is essentially a process of secondary segmentation and combination of the original multiple target connected regions.

[0082] Step 403: In response to the satisfaction of the iteration termination condition, determine the corresponding response sub-image based on each target connected region. It should be understood that the iteration termination condition here may include the fact that all multiple reference features have been traversed, that is, all multiple reference features of the response image have been used and the corresponding multiple target connected regions have been determined.

[0083] When the iteration termination condition is met, it indicates that all multiple reference features of the response image have been traversed. Therefore, the corresponding response sub-image can be determined based on each of the finally determined multiple target connected regions. Because multiple reference features are used in a sequential iteration manner, the segmentation result of the response image can be more accurate and is not affected by complex layout information.

[0084] In one alternative approach, the aforementioned multiple reference features may include shallow image features. If the target reference features include shallow image features, determining the multiple target connected regions contained in the answer image that match the target reference features may include: first, extracting the shallow image features contained in the answer image, and then fusing the shallow image features contained in the answer image based on connected component analysis to obtain multiple target connected regions.

[0085] In practical applications, the aforementioned shallow image features include color features, shape features, and texture features. These shallow image features can be extracted from the response image using the methods mentioned earlier. Then, based on connected component analysis, these shallow image features are fused. Here, the response image can be a single image or multiple adjacent response images. When fusing the response images, fusion can be performed first within a single response image, and then fusion can be performed between multiple adjacent response images to improve the accuracy of response image segmentation.

[0086] For example, the aforementioned shallow image features may include multiple target shallow image features. If the target reference features include target shallow image features, the shallow image features of the answer image are fused based on the connected component analysis method to obtain multiple target connected components. This may include: if two adjacent answer images contain the same target shallow image features, the two adjacent answer images are fused.

[0087] In practical applications, when the aforementioned shallow image features include color features, shape features, and texture features, the target shallow image features can be color features, shape features, or texture features. If the target shallow image feature is a color feature, and two adjacent response images contain the same color feature, then the two adjacent response images are fused.

[0088] In one alternative approach, when the aforementioned multiple reference features include deep semantic features, if the target reference feature is a deep semantic feature, the determination of multiple target connected regions in the answer image that match the target reference feature may include: first, extracting multiple text fragments included in the answer text contained in the answer image, and then determining the sentence relevance based on the target reference feature, fusing the multiple text fragments to obtain multiple target connected regions.

[0089] For example, the above-mentioned method of determining the statement relevance based on target reference features and fusing multiple text segments to obtain multiple target connected regions may include: if the statement relevance of two adjacent text segments is greater than or equal to a preset relevance, then fusing the adjacent response texts. It should be understood that the preset relevance can be set according to the actual situation and is not limited here.

[0090] In practical applications, multiple text segments containing the answer text in the answer image can be extracted using a neural network model, as described above. Then, the sentence correlation between multiple text segments can be determined based on deep semantic features. If the sentence correlation between two adjacent text segments is greater than or equal to the preset correlation, the adjacent answer texts are fused.

[0091] As one possible implementation, the matching method between the answer image and the question stem image in the exemplary embodiment of this disclosure may further include: when the answer sub-image is detected to have alteration features, deleting the answer sub-image with alteration features from among the multiple answer sub-images. It should be understood that the alteration features here may be alteration features such as lines, arrows, circles, etc.

[0092] For example, the presence of alteration features in the response sub-image can be detected using an object detection model. The response sub-image can be input into a trained object detection model to determine if alteration features exist. For instance, this object detection model can be a YOLOv5 object detection model, which is highly efficient and performs well in object detection. Forward inference is performed on the YOLOv5 object detection model based on the response sub-image data, and the model outputs the alteration features contained in the response sub-image. It should be understood that the object detection model here can also be the SSD object detection algorithm or the R-CenterNet object detection algorithm, etc. When the object detection model determines that alteration features exist in the response sub-image, the response sub-image containing alteration features is deleted from the multiple response sub-images.

[0093] For example, optical character recognition (OCR) technology can be used to identify multiple response sub-images and obtain the corresponding text information. Then, an image similarity matching algorithm is used to compare the text information corresponding to the multiple response sub-images to determine whether the corresponding response sub-images have been altered. If a response sub-image has been altered, the sub-images with alteration features are deleted from the multiple response sub-images. Then, a layout analysis model can be used to reposition the response sub-images and restore them to their original positions. Alternatively, a feature matching algorithm can be used to accurately locate the correct response sub-images, and then a region growing algorithm can be used to re-detect the correct response sub-images to replace and merge the altered areas.

[0094] The foregoing primarily describes the solutions provided by the embodiments of this disclosure from the perspective of the server. It is understood that, in order to implement the above functions, the server includes the corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, this disclosure can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0095] This disclosure embodiment can divide the server into functional units according to the above method example. For example, it can divide each function into a separate functional module, or it can integrate two or more functions into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this disclosure embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0096] In the case of dividing each functional module according to its corresponding functions, an exemplary embodiment of this disclosure provides a matching device for answer images and question stem images. The matching device for answer images and question stem images can be a server or a chip applied to a server. Figure 5 A schematic block diagram of the functional modules of a matching device for answer images and question stem images according to an exemplary embodiment of the present disclosure is shown. Figure 5 As shown, the matching device 500 for the answer image and the question stem image includes:

[0097] The extraction module 501 is used to extract multiple reference features of the answer image contained in the question image.

[0098] The module 502 is used to perform iterative feature fusion on the answer image using multiple reference features to obtain multiple answer sub-images. It also performs feature matching on the multiple question stem images and multiple answer sub-images contained in the question image to obtain the answer sub-image corresponding to each question stem image.

[0099] As one possible implementation, the above-mentioned obtaining module 502 is also used to perform layout analysis on the test question image to obtain the positioning information and type information of multiple sub-region images included in the test question image.

[0100] The matching device 500 for answer images and question stem images in an exemplary embodiment of this disclosure further includes an acquisition module 503. If the type information of multiple sub-region images matches the preset type information of the answer image, the acquisition module 503 is used to acquire the answer image from the question image based on the positioning information of the multiple sub-region images. If the type information of multiple sub-region images matches the preset type information of the question stem image, the acquisition module 503 is used to acquire the question stem image from the question image based on the positioning information of the multiple sub-region images.

[0101] In some alternative methods, the preset type of the above-mentioned answer image includes the font type of the answer image, and the preset type of the question stem image includes the font type of the question stem image.

[0102] As one possible implementation, the matching device 500 for the answer image and the question stem image in the exemplary embodiment of this disclosure further includes a determining module 504. The determining module 504 is used to determine multiple target connected regions contained in the answer image that match the target reference features. The preset feature values ​​of the reference features corresponding to the multiple target connected regions are different, and the target reference feature is one of the multiple reference features.

[0103] The matching device 500 for answer image and question stem image in an exemplary embodiment of this disclosure further includes an update module 505, which is used to update the answer image based on multiple target connected regions and update the target reference features based on multiple reference features in response to satisfying the iteration continuation condition, and to determine the corresponding answer sub-image based on each target connected region in response to satisfying the iteration termination condition.

[0104] In some alternative methods, the above-mentioned multiple reference features include shallow image features. If the target reference features include shallow image features, the extraction module 501 is also used to extract the shallow image features contained in the answer image.

[0105] The module 502 is also used to fuse the shallow image features contained in the answer image based on the connected component analysis method to obtain multiple target connected components.

[0106] In some alternative embodiments, the aforementioned shallow image features include multiple target shallow image features. If the target reference features include target shallow image features, the matching device 500 for answer images and question stem images in the exemplary embodiment of this disclosure further includes a fusion module 506. If two adjacent answer images contain the same target shallow image features, the fusion module 506 is used to fuse the two adjacent answer images.

[0107] In some alternative methods, the aforementioned multiple reference features include deep semantic features. If the target reference feature is a deep semantic feature, the extraction module 501 is used to extract multiple text fragments included in the answer text contained in the answer image, and the obtaining module 502 is used to determine the sentence relevance based on the target reference feature, fuse the multiple text fragments, and obtain multiple target connected regions.

[0108] In some optional methods, if the sentence correlation between two adjacent text segments is greater than or equal to the preset correlation, the above-mentioned fusion module 506 is used to fuse adjacent response texts.

[0109] As one possible implementation, the matching device 500 for the answer image and the question stem image in the exemplary embodiment of this disclosure further includes a deletion module 507. When the answer sub-image is detected to have alteration features, the deletion module 507 is used to delete the answer sub-image with alteration features from multiple answer features.

[0110] Figure 6 A schematic block diagram of a chip according to an exemplary embodiment of the present disclosure is shown. Figure 6 As shown, the chip 600 includes one or more (including two) processors 601 and a communication interface 602. The communication interface 602 can support the server in performing the data transmission and reception steps in the above-described image processing method, and the processor 601 can support the server in performing the data processing steps in the above-described image processing method.

[0111] Optional, such as Figure 6 As shown, the chip 600 also includes a memory 603, which may include read-only memory and random access memory, and provides operation instructions and data to the processor. A portion of the memory may also include non-volatile random access memory (NVRAM).

[0112] In some implementations, such as Figure 6As shown, processor 601 executes corresponding operations by calling operation instructions stored in memory (which may be stored in the operating system). Processor 601 controls the processing operations of any terminal device; processor can also be called a central processing unit (CPU). Memory 603 may include read-only memory and random access memory, and provides instructions and data to processor 601. A portion of memory 603 may also include NVRAM. For example, in applications, memory, communication interfaces, and other components are coupled together via a bus system, which may include, in addition to a data bus, a power bus, a control bus, and a status signal bus, etc. However, for clarity, in... Figure 6 The general designated all buses as Bus System 604.

[0113] The methods disclosed in the embodiments of this disclosure can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above methods can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, a digital signal processor (DSP), an ASIC, a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this disclosure. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this disclosure can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above methods.

[0114] Exemplary embodiments of this disclosure also provide an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to cause the electronic device to perform a method according to an embodiment of this disclosure.

[0115] Exemplary embodiments of this disclosure also provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a computer's processor, is used to cause the computer to perform a method according to embodiments of this disclosure.

[0116] Exemplary embodiments of this disclosure also provide a computer program product, including a computer program, wherein, when executed by a processor of a computer, the computer program is used to cause the computer to perform a method according to an embodiment of this disclosure.

[0117] refer to Figure 7 The present invention describes a structural block diagram of an electronic device 700 that can serve as a server or client of the present disclosure, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0118] like Figure 7 As shown, the electronic device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. The RAM 703 may also store various programs and data required for the operation of the electronic device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0119] Multiple components in electronic device 700 are connected to I / O interface 705, including: input unit 706, output unit 707, storage unit 708, and communication unit 709. Input unit 706 can be any type of device capable of inputting information to electronic device 700. Input unit 706 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of electronic device. Output unit 707 can be any type of device capable of presenting information and may include, but is not limited to, a display, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 708 may include, but is not limited to, disk and optical disk. Communication unit 709 allows electronic device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth™ devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0120] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above. For example, in some embodiments, the methods of exemplary embodiments of this disclosure can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 700 via ROM 702 and / or communication unit 709. In some embodiments, the computing unit 701 can be configured to perform the methods of exemplary embodiments of this disclosure by any other suitable means (e.g., by means of firmware).

[0121] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0122] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0123] As used in this disclosure, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0124] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0125] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0126] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

[0127] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed on a computer, the processes or functions described in the embodiments of this disclosure are performed, in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a terminal, a user equipment, or other programmable device. The computer program or instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center integrating one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; it can also be an optical medium, such as a digital video disc (DVD); or it can be a semiconductor medium, such as a solid-state drive (SSD).

[0128] Although this disclosure has been described in conjunction with specific features and embodiments, it will be apparent that various modifications and combinations can be made therein without departing from the spirit and scope of this disclosure. Accordingly, this specification and drawings are merely exemplary illustrations of the disclosure as defined by the appended claims and are to be considered as covering any and all modifications, variations, combinations, or equivalents within the scope of this disclosure. It is obvious that those skilled in the art can make various alterations and modifications to this disclosure without departing from its spirit and scope. Thus, this disclosure is also intended to include any such modifications and modifications that fall within the scope of the claims of this disclosure and their equivalents.

Claims

1. A method for matching answer images and question stem images, characterized in that, The method includes: Extracting question stem and answer images from question images; Extract multiple reference features from the response image; The answer image is identified as containing multiple target connected regions that match the target reference feature. The preset feature values ​​of the reference features corresponding to the multiple target connected regions are different, and the target reference feature is one of the multiple reference features. In response to satisfying the iteration continuation condition, the response image is updated based on the multiple target connected regions, and the target reference features are updated based on the multiple reference features; In response to the satisfaction of the iteration termination condition, a corresponding response sub-image is determined based on each of the target connected regions; Feature matching is performed on the multiple question stem images and multiple answer sub-images contained in the question image to obtain the answer sub-image corresponding to each question stem image.

2. The method according to claim 1, characterized in that, The method further includes: The layout of the test question image is analyzed to obtain the location and type information of multiple sub-region images included in the test question image; If the type information of multiple sub-region images matches the preset type information of the answer image, the answer image is obtained from the question image based on the positioning information of the multiple sub-region images; If the type information of multiple sub-region images matches the preset type information of the question stem image, the question stem image is obtained from the question image based on the positioning information of the multiple sub-region images.

3. The method according to claim 2, characterized in that, The preset type of the response image includes the font type of the response image; The preset type of the question image includes the font type of the question image.

4. The method according to claim 1, characterized in that, The plurality of reference features include shallow image features. If the target reference features include shallow image features, determining the plurality of target connected regions in the response image that match the target reference features includes: Extract the shallow image features contained in the response image; Based on connected component analysis, the shallow image features contained in the response image are fused to obtain multiple target connected components.

5. The method according to claim 4, characterized in that, The shallow image features include multiple target shallow image features. If the target reference features include the target shallow image features, the shallow image features of the response image are fused based on connected component analysis to obtain multiple target connected components, including: If two adjacent response images contain the same target shallow image features, the two adjacent response images are fused.

6. The method according to claim 3, characterized in that, The plurality of reference features include deep semantic features. If the target reference feature is a deep semantic feature, determining the plurality of target connected regions in the response image that match the target reference feature includes: Extract multiple text fragments from the response text contained in the response image; Based on the target reference features, the relevance of the sentences is determined, and multiple text segments are fused to obtain multiple target connected regions.

7. The method according to claim 6, characterized in that, The step of determining the sentence relevance based on the target reference features and fusing multiple text segments to obtain multiple target connected regions includes: If the sentence correlation between two adjacent text segments is greater than or equal to a preset correlation, the adjacent response texts are merged.

8. The method according to any one of claims 1-7, characterized in that, The method further includes: When the alteration feature is detected in the response sub-image, the response sub-image containing the alteration feature is deleted from the plurality of response sub-images.

9. A matching device for answer images and question stem images, characterized in that, The device includes: The extraction module is used to obtain the question stem image and the answer image based on the question image; The extraction module is also used to extract multiple reference features of the answer image contained in the question image; The determining module is used to determine multiple target connected regions in the response image that match the target reference features, wherein the preset feature values ​​of the reference features corresponding to the multiple target connected regions are different, and the target reference feature is one of the multiple reference features; The update module is used to update the response image based on multiple target connected regions and update the target reference features based on multiple reference features in response to the satisfaction of the iteration continuation condition; The update module is also configured to determine a corresponding response sub-image based on each of the target connected regions in response to the satisfaction of the iteration termination condition; The module is used to perform iterative feature fusion on the answer image using multiple reference features to obtain multiple answer sub-images, and to perform feature matching on multiple question stem images and multiple answer sub-images contained in the question image to obtain the answer sub-image corresponding to each question stem image.

10. An electronic device, characterized in that, include: processor; as well as, Memory for stored programs; The program includes instructions that, when executed by the processor, cause the processor to perform the method according to any one of claims 1-8.

11. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-8.

Images