A human-machine interface content recognition method, system, device and medium

Abstract

The application relates to a human-computer interface content recognition method, system, device and medium, which comprises the following steps: establishing a feature template database for acquired human-computer interface content based on a pre-constructed CNN-HMM hybrid model; performing one or more matching pre-processes such as ROI target region interception, pretreatment, threshold segmentation, correction and character cutting on the original image of the human-computer interface in a specified area; obtaining a confidence value by matching the image data obtained through the matching pre-process with the data in the constructed feature template database, and outputting a content recognition result according to the confidence value. The CNN end-to-end embedded HMM combines the strong recognition capability of the CNN and the sequence modeling capability of the HMM, and the accuracy and recognition rate of the constructed model are superior to those of the existing mature image recognition model in the case that the multiple human-computer interfaces in the experimental scene are easily affected by shielding and light.

Description

Technical Field

[0001] This invention relates to the field of image content recognition technology, and in particular to a method, system, device, and medium for human-computer interface content recognition. Background Technology

[0002] With the development of technology, modern laboratories have made great progress. In the daily testing experiments in the laboratory, most aspects such as the testing of instruments and equipment have been automated. However, the various instruments and equipment are scattered in the experimental area, which makes it difficult to record the experimental parameters of each device, and still requires manual input by the experimental personnel.

[0003] The presence of human intervention makes it impossible to centrally control all the equipment used in testing experiments, which greatly reduces processing efficiency. Consequently, the difficulty in recording experimental parameters for instruments and equipment located in fixed areas remains a major obstacle to further improving the automation level of modern laboratories. Summary of the Invention

[0004] (a) Technical problems to be solved

[0005] In view of the above-mentioned shortcomings and deficiencies of the prior art, the present invention provides a human-machine interface content recognition method, system, device and medium, which solves the technical problem that it is difficult to automatically record the test parameters of various instruments and equipment in the laboratory.

[0006] (II) Technical Solution

[0007] To achieve the above objectives, the main technical solutions adopted by the present invention include:

[0008] In a first aspect, embodiments of the present invention provide a method for human-computer interface content recognition based on a CNN / HMM hybrid model, comprising:

[0009] A feature template database is established based on a pre-built CNN-HMM hybrid model to acquire human-computer interface content;

[0010] Based on the CNN-HMM hybrid model, the original image of the human-computer interface in the specified region is subjected to one or more matching preprocessing steps, including ROI target region cropping, grayscale processing, threshold segmentation, correction, and character cutting.

[0011] The confidence score is obtained by matching the image data obtained after pre-matching with the feature templates in the feature template database, and the recognition result is output based on the obtained confidence score.

[0012] The CNN-HMM hybrid model is a model obtained by setting the output of the CNN layer and the observation probability matrix of the HMM layer end-to-end. In the CNN-HMM hybrid model, the CNN layer provides the output probability of any state to the HMM layer, and the HMM layer provides the classification of the corresponding state to the CNN layer.

[0013] Optionally, building a feature template database for the acquired human-computer interface content based on a pre-built CNN-HMM mixture model includes:

[0014] Based on the CNN layer of the pre-built CNN-HMM hybrid model, the historical original images of the human-computer interface are extracted to generate several feature sequences ordered by preset numbers. The feature styles of the data stored in each feature sequence are different.

[0015] The HMM layer of the pre-built CNN-HMM hybrid model models each feature sequence separately to obtain several sequence models. All sequence models are optimized and trained under the HMM layer to obtain a feature template database.

[0016] Optionally, the pre-processing of the original image of the human-computer interface in the specified region based on the CNN-HMM hybrid model includes one or more of the following: ROI target region cropping, grayscale processing, threshold segmentation, correction, and character cutting.

[0017] The robot moves a camera to acquire raw images of the human-computer interface in a designated area;

[0018] The region of interest (ROI) is extracted from the original image by using a CNN-HMM hybrid model.

[0019] A high-contrast grayscale image is obtained by extracting the minimum channel from the region of interest;

[0020] The high-contrast grayscale image is sequentially subjected to threshold segmentation, correction, and character cutting to complete the pre-matching processing.

[0021] Optionally, the confidence value is obtained by matching the image data obtained after pre-matching with the feature templates in the feature template database, and the recognition result is output based on the obtained confidence value, including:

[0022] The image data obtained after matching preprocessing is matched with each content feature data in the feature template database to obtain a confidence value;

[0023] When the confidence level is not less than 0.95, the recognition is considered successful, and the recognition result, including the confidence level and the corresponding feature template, is output to the test station with the pre-established communication connection.

[0024] When the confidence level is less than 0.95, it is judged as a recognition failure, and the recognition result containing the recognition failure information is pre-established with a test station for communication connection;

[0025] The test station outputs a test report based on the obtained recognition results.

[0026] Optionally, before building a feature template database for the acquired human-computer interface content based on a pre-built CNN-HMM hybrid model, the following steps are also included:

[0027] Data preprocessing is performed on the training set containing historical raw images of the acquired human-computer interface;

[0028] The CNN layer in the CNN-HMM hybrid model is used to classify the preprocessed data and output a posterior probability distribution with a sum of 1, that is, the probability of each character, button and graphic.

[0029] The obtained posterior probability distribution is input into the HMM layer of the CNN-HMM hybrid model to generate the initial probability and transition probability of the HMM layer.

[0030] Optionally, the HMM layer includes: a multi-state HMM structure and a single-state HMM structure;

[0031] The multi-state HMM structure is provided with multiple state units that correspond one-to-one with the different appearances of the characters.

[0032] The single-state HMM structure is provided with a jump connection unit for connecting two adjacent characters.

[0033] Optionally, the window size of the downsampling layers in the CNN layer is 2×2, and all nonlinear units in the CNN layer use ReLU units.

[0034] Secondly, embodiments of the present invention provide a human-computer interface content recognition system based on a CNN / HMM hybrid model, comprising:

[0035] A mobile camera robot is used to acquire raw images of the human-machine interface in a designated area along a preset patrol path according to pre-configured movement instructions.

[0036] The feature template database establishment module is used to establish a feature template database for the acquired human-computer interface content based on a pre-built CNN-HMM hybrid model.

[0037] The matching preprocessing module is used to perform one or more of the following matching preprocessing steps on the original image of the human-computer interface in the specified region based on the CNN-HMM hybrid model: region of interest cropping, preprocessing, threshold segmentation, and character cutting.

[0038] The content recognition module is used to obtain a confidence value by matching the image data obtained after matching preprocessing with the data in the constructed feature template database, and output the content recognition result based on the obtained confidence value;

[0039] The CNN-HMM hybrid model is a model obtained by setting the output of the CNN layer and the observation probability matrix of the HMM layer end-to-end. In the CNN-HMM hybrid model, the CNN layer provides the output probability of any state to the HMM layer, and the HMM layer provides the classification of the corresponding state to the CNN layer.

[0040] Thirdly, according to an embodiment of the present invention, a human-computer interface content recognition device based on a CNN / HMM hybrid model includes:

[0041] At least one database, the database storing original images of human-computer interfaces in a preset area and corresponding feature sequences;

[0042] and a memory that is communicatively connected to the at least one database;

[0043] The memory stores instructions that can be executed by the at least one database, which are then executed by the at least one database to enable the at least one database to perform a human-computer interface content recognition method based on a CNN / HMM hybrid model as described above.

[0044] Fourthly, an embodiment of the present invention provides a computer-readable medium having computer-executable instructions stored thereon, wherein when the executable instructions are executed by a processor, they implement the human-computer interface content recognition method based on a CNN / HMM hybrid model as described above.

[0045] (III) Beneficial Effects

[0046] The beneficial effects of this invention are as follows: This invention embeds a CNN end-to-end into an HMM to construct a CNN / HMM hybrid model. This hybrid model combines the strong recognition capabilities of CNNs with the sequence modeling capabilities of HMMs. Even in experimental scenarios where multi-person computer interfaces are susceptible to occlusion and lighting conditions, its accuracy and recognition speed outperform existing mature image recognition models such as AlexNet, VGG19, and ResNet_152. Simultaneously, this invention solves the problem of recording experimental parameters from various instruments and equipment scattered throughout the experimental area, greatly improving the level of laboratory automation and possessing practical application value. Attached Figure Description

[0047] Figure 1 A flowchart illustrating a human-computer interface content recognition method based on a CNN / HMM hybrid model provided in an embodiment of the present invention;

[0048] Figure 2 A schematic diagram of a CNN-HMM hybrid model for a robot walking operation method based on robot vision provided in an embodiment of the present invention;

[0049] Figure 3 A detailed flowchart illustrating step S1 of a human-computer interface content recognition method based on a CNN / HMM hybrid model provided in this embodiment of the invention;

[0050] Figure 4 A detailed flowchart illustrating step S2 of a human-computer interface content recognition method based on a CNN / HMM hybrid model provided in this embodiment of the invention;

[0051] Figure 5 A detailed flowchart illustrating step S3 of a human-computer interface content recognition method based on a CNN / HMM hybrid model provided in this embodiment of the invention;

[0052] Figure 6 The original image of the human-computer interface for a human-computer interface content recognition method based on a CNN / HMM hybrid model provided in this embodiment of the invention;

[0053] Figure 7 An image processed by ROI cropping and grayscale conversion, provided as an embodiment of the present invention, for a human-computer interface content recognition method based on a CNN / HMM hybrid model;

[0054] Figure 8 A threshold-segmented image is provided as an embodiment of the present invention for a human-computer interface content recognition method based on a CNN / HMM hybrid model.

[0055] Figure 9 A corrected image provided by an embodiment of the present invention for a human-computer interface content recognition method based on a CNN / HMM hybrid model;

[0056] Figure 10 The image segmented by characters is provided as an embodiment of the present invention for a human-computer interface content recognition method based on a CNN / HMM hybrid model;

[0057] Figure 11 The confidence level of a character-segmented image in a human-computer interface content recognition method based on a CNN / HMM hybrid model provided in this embodiment of the invention;

[0058] Figure 12 The flowchart illustrates the process of comprehensively judging the recognition results of deep learning based on the CNN-HMM hybrid model and the recognition results of traditional image processing in a human-computer interface content recognition method based on the CNN / HMM hybrid model provided in this embodiment of the invention.

[0059] Figure 13 The training process of a CNN-HMM hybrid model for a human-computer interface content recognition method based on a CNN / HMM hybrid model provided in this embodiment of the invention;

[0060] Figure 14 A schematic diagram of the HMM layer structure of a human-computer interface content recognition method based on a CNN / HMM hybrid model provided in an embodiment of the present invention;

[0061] Figure 15 A schematic diagram of the state merging of the HMM layer in a human-computer interface content recognition method based on a CNN / HMM hybrid model provided in an embodiment of the present invention;

[0062] Figure 16 A schematic diagram of a text-containing button provided in an embodiment of the present invention for a human-computer interface content recognition method based on a CNN / HMM hybrid model;

[0063] Figure 17 Another schematic diagram of a text-containing button, provided as an embodiment of the present invention, illustrates a human-computer interface content recognition method based on a CNN / HMM hybrid model. Detailed Implementation

[0064] To better explain and facilitate understanding of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0065] like Figure 1 As shown in the figure, an embodiment of the present invention proposes a method for human-computer interface content recognition based on a CNN / HMM hybrid model, comprising: First, establishing a feature template database for human-computer interface content containing characters, buttons, and graphics based on a pre-built CNN-HMM hybrid model; Second, performing one or more matching preprocessing steps on the original image of the human-computer interface in a specified region, including ROI target region cropping, grayscale processing, threshold segmentation, correction, and character cutting; Third, obtaining a confidence value by matching the image data obtained after matching preprocessing with the data in the feature template database, and outputting the content recognition result based on the obtained confidence value; wherein, as... Figure 2 As shown, the CNN-HMM hybrid model is a model obtained by setting the output of the CNN layer and the observation probability matrix of the HMM layer end-to-end. In the CNN-HMM hybrid model, the CNN layer provides the output probability of any state to the HMM layer, and the HMM layer provides the classification of the corresponding state to the CNN layer.

[0066] This invention embeds a CNN end-to-end into a Hidden Model (HMM) to construct a CNN / HMM hybrid model. This hybrid model combines the strong recognition capabilities of CNNs with the sequence modeling capabilities of HMMs. Even in multi-person computer interfaces in experimental scenarios where occlusion and lighting conditions can affect performance, its accuracy and recognition speed outperform existing mature image recognition models such as AlexNet, VGG19, and ResNet_152. Furthermore, this invention solves the problem of recording experimental parameters from various instruments and equipment scattered throughout the experimental area, significantly improving laboratory automation and possessing practical application value.

[0067] To better understand the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present invention can be understood more clearly and thoroughly, and that the scope of the present invention can be fully conveyed to those skilled in the art.

[0068] Specifically, the present invention provides a method for human-computer interface content recognition based on a CNN / HMM hybrid model, which includes:

[0069] S1. Based on the pre-built CNN-HMM hybrid model, establish a feature template database for human-computer interface content containing characters, buttons, and graphics.

[0070] Furthermore, such as Figure 3 As shown, step S1 includes:

[0071] S11. Based on the CNN layer of the pre-built CNN-HMM hybrid model, the historical original images of the human-computer interface are used for feature extraction, generating several feature sequences sorted by preset numbers. The feature styles of the saved data in each feature sequence are different.

[0072] S12. The HMM layer based on the pre-built CNN-HMM hybrid model models each feature sequence separately to obtain several sequence models. All sequence models are optimized and trained under the HMM layer to obtain a feature template database.

[0073] In one specific embodiment, the CNN layer of a pre-built CNN-HMM hybrid model extracts features from the original image containing the text content recognition of "success" and "failure" to construct a feature sequence. This can be simply understood as saving the various display features of the word "success" on the screen to the same folder and sorting them by number. Then, the HMM layer of the pre-built CNN-HMM hybrid model models the feature sequences of "success" and "failure" respectively. The sequence modeling is optimized and trained under the HMM layer to obtain a database containing feature templates of "success" and "failure" content respectively.

[0074] S2. Perform one or more of the following pre-processing steps on the original image of the human-computer interface in the specified area: ROI target region cropping, grayscale processing, threshold segmentation, correction, and character cutting.

[0075] Furthermore, such as Figure 4 As shown, step S2 includes:

[0076] S21. Obtain the original image of the human-machine interface in the specified area by moving the camera robot.

[0077] S22. Based on a preset deep learning model, the ROI target region is extracted from the acquired original image to obtain the region of interest.

[0078] S23. Obtain a high-contrast grayscale image by extracting the minimum channel from the region of interest.

[0079] S24. The grayscale image with strong contrast is subjected to threshold segmentation, correction and character cutting in sequence to complete the pre-matching processing operation.

[0080] S3. The confidence value is obtained by matching the image data obtained after pre-matching with the feature templates in the feature template database, and the content recognition result is output based on the obtained confidence value.

[0081] Furthermore, such as Figure 5 As shown, step S3 includes:

[0082] S31. The image data obtained after matching preprocessing is matched with each content feature data in the feature template database to obtain a confidence value.

[0083] S32. When the confidence level is not less than 0.95, the recognition is considered successful, and the recognition result, including the confidence level and the corresponding feature template, is output to the test station with the pre-established communication connection.

[0084] S33. When the confidence level is less than 0.95, it is judged as recognition failure, and the test station with the recognition result containing recognition failure information is pre-established with a communication connection.

[0085] S34. The test station outputs a test report based on the obtained recognition results.

[0086] refer to Figures 6-11 It can be seen that when the human-machine interface recognition is officially performed, the content of the image taken by the industrial camera is sequentially processed by cropping the ROI region, grayscale conversion, threshold segmentation, correction, character cutting, and matching and comparison with the content feature library to obtain the confidence value (i.e. the similarity of the match). When the confidence is greater than or equal to 0.95, the recognition result is directly output to the test station. When the confidence is less than 0.95, the result is output to the test station according to the set prompt, such as "recognition failed". Finally, the test station outputs a test report.

[0087] Ideally, the accuracy of character segmentation can be assessed: by statistically analyzing the height and width of the segmented characters, if the difference between the height and width of the segmented characters exceeds a certain threshold, an error message indicating abnormal character segmentation can be displayed.

[0088] Next, refer to Figure 12 It is understood that a comprehensive judgment result can also be output based on the recognition results of deep learning of the CNN-HMM hybrid model provided by this invention and the recognition results of traditional image processing, so as to improve accuracy.

[0089] And, before step S1, as Figure 13 As shown, it also includes:

[0090] F11. Perform data preprocessing on the training set containing the acquired historical raw images of the human-computer interface.

[0091] F12. Use the CNN layer in the CNN-HMM hybrid model to classify the preprocessed data and output a posterior probability distribution with a sum of 1, that is, the probability of each character, button and graphic.

[0092] F13. Input the obtained posterior probability distribution into the HMM layer of the CNN-HMM hybrid model to generate the initial probability and transition probability of the HMM layer.

[0093] Furthermore, such as Figure 14 As shown, HMM contains two topologies. Figure 14 The structure of the HMM for the left character is uniformly adopted as a 3-state HMM from left to right, which models the front, middle and back appearances of the character respectively. Figure 7 The corresponding HMM structure on the right is a single-state HMM. This structure also includes jump connections to accommodate cases where two characters are closely connected and the blank model can be omitted. Figure 14 Two HMM structures are shown: hollow circles represent hidden states, solid circles represent visible states, and squares represent entry and exit states that do not generate observations.

[0094] The three states mentioned above allow for a more detailed classification of character appearance. However, for discriminative models like CNNs, especially given their strong discriminative power, classifying different states within an HMM model into different categories can interfere with the CNN's function. Therefore, when using a CNN-HMM hybrid model, the states within an HMM model are merged. The merged states then represent the categories that the CNN model needs to classify. Figure 15 This demonstrates the principle behind the merger.

[0095] Furthermore, the CNN layer adopts the existing mature CNN model structure, specifically configured as follows: the window size of the downsampling layer is 2×2, and all nonlinear units use ReLU units.

[0096] Furthermore, this invention also provides a human-computer interface content recognition system based on a CNN / HMM hybrid model, which includes:

[0097] A mobile camera robot is used to acquire raw images of the human-machine interface in a designated area along a preset patrol path according to pre-configured movement instructions.

[0098] The feature template database creation module is used to create a feature template database for human-computer interface content containing characters, buttons, and graphics based on a pre-built CNN-HMM hybrid model.

[0099] The matching preprocessing module is used to perform one or more of the following matching preprocessing steps on the original image of the human-computer interface in the specified area: cropping the region of interest, preprocessing, threshold segmentation, and character cutting.

[0100] The content recognition module is used to obtain a confidence value by matching the image data obtained after matching preprocessing with the data in the constructed feature template database, and output the content recognition result based on the obtained confidence value;

[0101] The CNN-HMM hybrid model is a model obtained by setting the output of the CNN layer and the observation probability matrix of the HMM layer end-to-end. In the CNN-HMM hybrid model, the CNN layer provides the output probability of any state to the HMM layer, and the HMM layer provides the classification of the corresponding state to the CNN layer.

[0102] Meanwhile, the present invention also provides a human-computer interface content recognition device based on a CNN / HMM hybrid model, comprising: at least one database storing original images of a human-computer interface in a preset region and corresponding feature sequences; and a memory communicatively connected to the at least one database; wherein the memory stores instructions executable by the at least one database, the instructions being executed by the at least one database to enable the at least one database to perform a human-computer interface content recognition method based on a CNN / HMM hybrid model as described above.

[0103] Furthermore, the present invention also provides a computer-readable medium having computer-executable instructions stored thereon, wherein the executable instructions, when executed by a processor, implement the human-computer interface content recognition method based on a CNN / HMM hybrid model as described above.

[0104] In summary, this invention provides a method, system, device, and medium for human-computer interface content recognition. The CNN / HMM hybrid model proposed in this embodiment first uses a CNN layer to extract features from text images, constructing a feature sequence. Then, a Hidden Markov Model (HMM) is used for sequence modeling. The entire model is optimized and trained within the HMM framework without requiring explicit character segmentation. The overall structure of CNN-HMM is shown in the figure below, where the CNN represents character features and serves as the output model for observation vectors (i.e., modeling), while the HMM model is used to model the correlation of the feature sequence.

[0105] like Figure 16 and Figure 17 As shown, by using the CNN / HMM hybrid model provided by this invention, the human-machine interface of various equipment in the laboratory, which includes text, images and buttons containing text, can be quickly and accurately identified. This greatly accelerates the process of unmanned and automated laboratory operations, which is especially beneficial in harsh experimental environments such as dust-free, hazardous gas or material environments.

[0106] Since the systems / devices described in the above embodiments of the present invention are systems / devices used to implement the methods of the above embodiments of the present invention, those skilled in the art can understand the specific structure and modifications of the systems / devices based on the methods described in the above embodiments of the present invention, and therefore will not be repeated here. All systems / devices used in the methods of the above embodiments of the present invention fall within the scope of protection of the present invention.

[0107] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0108] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions.

[0109] It should be noted that any reference numerals placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In claims that enumerate several means, several of these means may be embodied by the same hardware. The use of the terms first, second, third, etc., is merely for convenience of expression and does not indicate any order. These terms can be understood as part of the component names.

[0110] Furthermore, it should be noted that in the description of this specification, the terms "one embodiment," "some embodiments," "embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0111] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the claims should be interpreted to include both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0112] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, then this invention should also include these modifications and variations.

Claims

1. A method for human-computer interface content recognition based on a CNN / HMM hybrid model, characterized in that, include: Data preprocessing is performed on the training set containing historical raw images of the acquired human-computer interface; The CNN layer in the CNN-HMM hybrid model is used to classify the preprocessed data and output a posterior probability distribution with a sum of 1, which serves as the probability of each character, button, and graphic. The obtained posterior probability distribution is input into the HMM layer of the CNN-HMM hybrid model to generate the initial probability and transition probability of the HMM layer; wherein, the HMM layer includes: a multi-state HMM structure and a single-state HMM structure; the multi-state HMM structure is provided with multiple state units that correspond one-to-one with different appearances of characters; the single-state HMM structure is provided with jump connection units for connecting two adjacent characters. A feature template database is established for the acquired human-computer interface content based on a pre-built CNN-HMM hybrid model. This includes: extracting features from historical original images of the human-computer interface acquired by the CNN layer of the pre-built CNN-HMM hybrid model, generating several feature sequences sorted by preset numbers, with each feature sequence having a different feature style; modeling each feature sequence separately using the HMM layer of the pre-built CNN-HMM hybrid model to obtain several sequence models, and optimizing and training all sequence models under the HMM layer to obtain the feature template database. Based on the CNN-HMM hybrid model, the original image of the human-computer interface in the specified region is subjected to one or more matching preprocessing steps, including ROI target region cropping, grayscale processing, threshold segmentation, correction, and character cutting. The confidence score is obtained by matching the image data obtained after pre-matching with the feature templates in the feature template database, and the recognition result is output based on the obtained confidence score. The CNN-HMM hybrid model is a model obtained by setting the output of the CNN layer and the observation probability matrix of the HMM layer end-to-end. In the CNN-HMM hybrid model, the CNN layer provides the output probability of any state to the HMM layer, and the HMM layer provides the classification of the corresponding state to the CNN layer.

2. The method for human-computer interface content recognition based on a CNN / HMM hybrid model as described in claim 1, characterized in that, Pre-processing based on the CNN-HMM hybrid model for acquiring the original image of the human-computer interface in a specified region includes one or more of the following: ROI target region cropping, grayscale processing, threshold segmentation, correction, and character cutting. The robot moves a camera to acquire raw images of the human-computer interface in a designated area; The region of interest (ROI) is extracted from the original image by using a CNN-HMM hybrid model. A high-contrast grayscale image is obtained by extracting the minimum channel from the region of interest; The high-contrast grayscale image is sequentially subjected to threshold segmentation, correction, and character cutting to complete the pre-matching processing.

3. The method for human-computer interface content recognition based on a CNN / HMM hybrid model as described in claim 1, characterized in that, The confidence score is obtained by matching the image data obtained after pre-processing with feature templates in the feature template database, and the recognition result is output based on the obtained confidence score, including: The image data obtained after matching preprocessing is matched with each content feature data in the feature template database to obtain a confidence value; When the confidence level is not less than 0.95, the recognition is considered successful, and the recognition result, including the confidence level and the corresponding feature template, is output to the test station with the pre-established communication connection. When the confidence level is less than 0.95, it is judged as a recognition failure, and the recognition result containing the recognition failure information is pre-established with a test station for communication connection; The test station outputs a test report based on the obtained recognition results.

4. A method for human-computer interface content recognition based on a CNN / HMM hybrid model as described in any one of claims 1-3, characterized in that, The downsampling layers in the CNN layer all have a window size of 2×2, and all nonlinear units in the CNN layer use ReLU units.

5. A human-computer interface content recognition system based on a CNN / HMM hybrid model, employing the method described in any one of claims 1-4, characterized in that, include: A mobile camera robot is used to acquire raw images of the human-machine interface in a designated area along a preset patrol path according to pre-configured movement instructions. The feature template database establishment module is used to establish a feature template database for the acquired human-computer interface content based on a pre-built CNN-HMM hybrid model. The matching preprocessing module is used to perform one or more of the following matching preprocessing steps on the original image of the human-computer interface in the specified region based on the CNN-HMM hybrid model: region of interest cropping, preprocessing, threshold segmentation, and character cutting. The content recognition module is used to obtain a confidence value by matching the image data obtained after matching preprocessing with the data in the constructed feature template database, and output the content recognition result based on the obtained confidence value; The CNN-HMM hybrid model is a model obtained by setting the output of the CNN layer and the observation probability matrix of the HMM layer end-to-end. In the CNN-HMM hybrid model, the CNN layer provides the output probability of any state to the HMM layer, and the HMM layer provides the classification of the corresponding state to the CNN layer.

6. A human-computer interface content recognition device based on a CNN / HMM hybrid model, characterized in that, include: At least one database, the database storing original images of human-computer interfaces in a preset area and corresponding feature sequences; And a memory that is communicatively connected to the at least one database; The memory stores instructions that can be executed by the at least one database, which are executed by the at least one database to enable the at least one database to perform a human-computer interface content recognition method based on a CNN / HMM hybrid model as described in any one of claims 1-4.

7. A computer-readable medium having computer-executable instructions stored thereon, characterized in that, When the executable instructions are executed by the processor, they implement a human-computer interface content recognition method based on a CNN / HMM hybrid model as described in any one of claims 1-4.

Images