Printing state detection method and device, electronic equipment, medium and program product

Abstract

The invention provides a printing state detection method which can be applied to the technical field of artificial intelligence. The printing state detection method comprises the steps that monitoring image data, log information, audio information and paper vibration information generated in the printing process are acquired; determining a paper state in the printing process from the monitoring image data based on a paper detection model; identifying text information in the audio information; analyzing the log information to obtain an initial printing state in the printing process; forming a paper vibration curve of the printing process based on the paper vibration information; based on a preset multi-modal model, performing feature fusion on the paper state, the text information, the initial printing state and the paper vibration curve to obtain a target fusion feature; and detecting a target printing state in the printing process according to the target fusion feature. The invention further provides a printing state detection device, electronic equipment, a medium and a program product.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and more particularly to a printing state detection method and device, electronic equipment, medium and program product. BACKGROUND

[0002] With the continuous acceleration of digital transformation, front desk business increasingly relies on the collaborative operation of various intelligent hardware peripherals, such as printers, signature screens, etc. These peripherals play a key role in the front desk system, and any hardware and software compatibility problems, functional failures or collaboration abnormalities will directly affect business efficiency and customer experience.

[0003] At present, most test system methods generally analyze multi-source data through intelligent algorithms, which cannot accurately identify hidden faults and subtle abnormalities, rely on the experience of test personnel, and are prone to misjudgment or omission. The peripheral test platform often relies on the logs returned by the device or the return value of the test interface, and cannot obtain the physical state (such as vibration, lag) during the running of the device, the image state (such as deviation, empty printing) or the voice broadcast situation in real time, resulting in incomplete and incomplete test information. SUMMARY

[0004] In view of the above problems, the present application provides a printing state detection method and device, electronic equipment, medium and program product, which uses multi-modal data such as images, logs, audio, vibration signals, etc. to realize the all-round collection and monitoring of device state, solving the problems of incomplete and incomplete test information and misjudgment.

[0005] According to a first aspect of the present application, a printing state detection method is provided, comprising: obtaining monitoring image data, log information, audio information and paper vibration information generated in a printing process; determining the paper state in the printing process from the monitoring image data based on a paper detection model; identifying the text information in the audio information; analyzing the log information to obtain the initial printing state in the printing process; forming a paper vibration curve of the printing process based on the paper vibration information; based on a preset multi-modal model, performing feature fusion on the paper state, the text information, the initial printing state and the paper vibration curve to obtain a fusion feature; and detecting the target printing state of the printing process according to the fusion feature.

[0006] According to an embodiment of the present application, identifying the text information in the audio information comprises: when the noise of the audio information is higher than a preset noise threshold, performing noise reduction processing on the audio information using a spectral subtraction method; and based on a speech recognition model, identifying the noise-reduced audio information to obtain the text information.

[0007] According to an embodiment of the present application, the multi-modal model comprises a convolution layer, an image encoder, a Transformer layer, a modal alignment layer, a multi-modal fusion layer, and a fusion discrimination layer; the feature fusion of the paper state, the text information, the initial printing state, and the paper vibration curve comprises: performing convolution processing on the paper state by using the convolution layer to obtain paper features, the paper features comprising a paper edge contour and a paper flatness; performing encoding on the paper features by using the image encoder to obtain a paper encoding vector; performing sorting on the text information and the initial printing state by using the Transformer layer to obtain a text vector and a state vector; aligning the text vector and the state vector by using the modal alignment layer to obtain an alignment vector; performing feature fusion on the paper encoding vector, the alignment vector, and the paper vibration curve by using the multi-modal fusion layer to obtain initial fusion features; and performing discrimination on the initial fusion features by using the fusion discrimination layer to obtain target fusion features, the target fusion features representing probabilities of each of a plurality of preset printing states.

[0008] According to an embodiment of the present application, the text information comprises a plurality of words, and the sorting of the text information by using the Transformer layer to obtain a text vector comprises: converting the plurality of words into a plurality of word vectors; obtaining time sequence features of the word vectors, the time sequence features representing an appearance order of the plurality of words in a printing process; sorting the plurality of word vectors according to the time sequence features, and determining an abnormal word vector that satisfies a preset association condition from the plurality of word vectors according to a sorting result. The abnormal word vector in the plurality of word vectors is removed, and the plurality of word vectors after the removal are taken as the text vector.

[0009] According to an embodiment of the present application, the determining of the abnormal word vector that satisfies the preset association condition from the plurality of word vectors according to the sorting result comprises: determining, from the plurality of word vectors, any word vector whose association degree with all adjacent word vectors is less than a preset association threshold according to the sorting result, and taking the word vector as the abnormal word vector.

[0010] According to an embodiment of the present application, the text vector includes a plurality of first sub-vectors, and the state vector includes a plurality of second sub-vectors; the text vector and the state vector are aligned by using a modal alignment layer to obtain an aligned vector, including: constructing a semantic matrix, the semantic matrix taking the plurality of first sub-vectors as rows and the plurality of second sub-vectors as columns; determining the similarity between each first sub-vector and each second sub-vector by using a cosine similarity algorithm, and filling the similarity into the corresponding position of the semantic matrix as an element of the semantic matrix; selecting elements with a similarity greater than a preset similarity threshold from the semantic matrix as a plurality of target elements, and determining the row index and the column index where each target element is located; determining a target first sub-vector corresponding to the row index from the plurality of first sub-vectors, and determining a target second sub-vector corresponding to the column index from the plurality of second sub-vectors, and combining the target first sub-vector and the target second sub-vector to form an aligned sub-vector; determining the weight of the plurality of aligned sub-vectors according to the similarity; and splicing the plurality of aligned sub-vectors according to the weight to form the aligned vector.

[0011] According to an embodiment of the present application, the initial fusion feature includes a plurality of sub-features, and the initial fusion feature is discriminated by using a fusion discrimination layer to obtain a target fusion feature, including: determining the association degree of each sub-feature with each preset printing state, the plurality of preset printing states including normal printing, paper jamming, and insufficient ink; when the association degree of any target sub-feature in the plurality of sub-features with any target preset printing state in the plurality of preset printing states is higher than a preset association degree threshold, the target sub-feature is classified into the target preset printing state, a feature set of each preset printing state is obtained, and the plurality of feature sets are taken as the target fusion feature.

[0012] According to an embodiment of the present application, the target printing state of the printing process is detected according to the target fusion feature, including: determining the probability of the plurality of feature sets according to the number of sub-features in the feature set, the probability representing the ratio of the number of sub-features in each feature set to the number of sub-features; determining the preset printing state with the highest probability in the plurality of preset printing states as the target printing state.

[0013] A second aspect of the present application provides a printing state detection device, including: a data acquisition module configured to acquire monitoring image data, log information, audio information, and paper vibration information generated in a printing process; a data processing module configured to determine a paper state in the printing process from the monitoring image data based on a paper detection model; identify text information in the audio information; analyze the log information to obtain an initial printing state in the printing process; form a paper vibration curve of the printing process based on the paper vibration information; a feature fusion module configured to perform feature fusion on the paper state, the text information, the initial printing state, and the paper vibration curve based on a preset multi-modal model to obtain a target fusion feature; and a state detection module configured to detect a target printing state of the printing process according to the target fusion feature.

[0014] The third aspect of the present application provides an electronic device, comprising: one or more processors; a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method.

[0015] The fourth aspect of the present application also provides a computer-readable storage medium having stored thereon a computer program or instructions, which, when executed by a processor, implement the steps of the method.

[0016] The fifth aspect of the present application also provides a computer program product comprising a computer program or instructions, which, when executed by a processor, implement the steps of the method. BRIEF DESCRIPTION OF DRAWINGS

[0017] The above content of the present application and other purposes, features and advantages will be more apparent through the following description of the embodiments of the present application with reference to the accompanying drawings, in which:

[0018] Figure 1 An application scenario diagram of the printing state detection method, device, electronic device, medium and program product according to the embodiments of the present application is schematically shown;

[0019] Figure 2 A flowchart of the printing state detection method according to the embodiments of the present application is schematically shown;

[0020] Figure 3 A flowchart of feature fusion according to the embodiments of the present application is schematically shown;

[0021] Figure 4 A flowchart of alignment of the text vector and the state vector according to the embodiments of the present application is schematically shown;

[0022] Figure 5 A structural block diagram of the printing state detection device according to the embodiments of the present application is schematically shown; and

[0023] Figure 6 A block diagram of an electronic device suitable for implementing the printing state detection method according to the embodiments of the present application is schematically shown. DETAILED DESCRIPTION

[0024] Embodiments of the present application will be described below with reference to the accompanying drawings. However, it should be understood that the description is merely exemplary and is not intended to limit the scope of the present application. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to one skilled in the art that one or more embodiments can be practiced without these specific details. In addition, in the following description, descriptions of well-known structures and techniques have been omitted to avoid unnecessarily obscuring the concepts of the present application.

[0025] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used herein, the term "includes" and tautological expressions thereof, means the inclusion of the features, steps, operations, and / or components listed but not to the exclusion of any others.

[0026] All terms used herein including technical and scientific terms have the same meaning as commonly understood by one of ordinary skill in the art unless otherwise defined herein. It should be noted that the terms used herein should be interpreted as having a meaning that is consistent with the context of the specification, and not be interpreted in an idealized or overly formal way.

[0027] In the case of using expressions similar to "at least one of A, B, and C, etc.", it should generally be interpreted to include at least one of A, B, or C, or a combination thereof (e.g., a system having at least one of A, B, and C should include but not be limited to a system having A alone, a system having B alone, a system having C alone, a system having both A and B, a system having both A and C, a system having both B and C, and / or a system having A, B, and C, etc.).

[0028] In the technical solutions of the present application, the user information (including but not limited to user personal information, user image information, user equipment information, such as location information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved are information and data authorized by the user or authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of related data comply with relevant laws, regulations and standards, necessary security measures are taken, do not violate public order and good customs, and provide corresponding operation portal for user to choose authorization or refusal.

[0029] In the scenario of making automated decisions by using personal information, the method, device and system provided by the embodiments of the present application all provide corresponding operation entrances for the user to select to agree or reject the automated decision result; if the user selects to reject, the expert decision process is entered. The expression "automated decision" here refers to the activity of making decisions by automatically analyzing and evaluating the personal behavior habits, interests and hobbies, or economic, health and credit conditions, etc. by a computer program. The expression "expert decision" here refers to the activity of making decisions by personnel who are engaged in a certain field of work, have special experience, knowledge and skills and reach a certain professional level.

[0030] Figure 1 An application scenario diagram of the printing state detection method, apparatus, electronic device, medium and program product according to the embodiments of the present application is schematically shown.

[0031] As shown in Figure 1 The application scenario 100 according to the embodiments can include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104 and a server 105. The network 104 is a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103 and the server 105. The network 104 can include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0032] The user can use the first terminal device 101, the second terminal device 102 and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102 and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0033] The first terminal device 101, the second terminal device 102 and the third terminal device 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers and desktop computers, etc.

[0034] The server 105 can be a server providing various services, such as a background management server supporting the website browsed by the user using the first terminal device 101, the second terminal device 102 and the third terminal device 103 (only as an example). The background management server can analyze and process the received user request data, etc., and feed back the processing result (such as a web page, information or data, etc. obtained or generated according to the user request) to the terminal device.

[0035] It should be noted that the printing state detection method provided in the embodiments of the present application can be generally executed by the server 105. Correspondingly, the printing state detection apparatus provided in the embodiments of the present application can be generally arranged in the server 105. The printing state detection method provided in the embodiments of the present application can also be executed by a server or a server cluster different from the server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Correspondingly, the printing state detection apparatus provided in the embodiments of the present application can also be arranged in a server or a server cluster different from the server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0036] It should be understood that Figure 1 The number of terminal devices, networks and servers in the above-mentioned scenario is only illustrative. According to the implementation needs, there can be any number of terminal devices, networks and servers.

[0037] The printing state detection method according to the embodiments of the present application will be described in detail below based on the scenario described above. Figure 1 Figures 2-4 The printing state detection method according to the embodiments of the present application will be described in detail below based on the scenario described above.

[0038] Figure 2 The flowchart of the printing state detection method according to the embodiments of the present application is schematically shown.

[0039] As shown in Figure 2 The printing state detection method according to the embodiments of the present application includes operations S210-S240.

[0040] In operation S210, the monitoring image data, log information, audio information and paper vibration information generated in the printing process are acquired.

[0041] The printing process can be the complete process from the user initiating a printing instruction to finally presenting the required content on the medium such as paper. When detecting the printing process, it is necessary to detect the abnormal situations of the printer cartridge or toner residual amount, paper jam, nozzle blockage and paper offset, etc. These situations can be reflected by the “log information”, “audio information” or “paper vibration information”.

[0042] For example, the monitoring image data can be image data acquired by shooting the printing process through an imaging device such as a camera, which can intuitively present the picture situation during printing, such as the display of the printed content, the position of the paper, etc., for subsequent judgment of paper problems, offset, empty printing, paper jam, etc.

[0043] ​For example, the log information can be interface call logs returned by a printer driver, device status codes, exception codes, etc., or operation, status, etc. information automatically recorded by a printing device during operation, containing the initiation time of a printing task, the running state change of the device, etc.

[0044] For example, the audio information can be sound data generated in the printing process collected by a recording device, such as mechanical sound when the printer is running, sound of paper movement, printing; or voice to be played after printing is completed, such as “printing is completed, please take the bill” and the like.

[0045] For example, the paper vibration information can be related data of paper vibration in the printing process captured by a sensor or the like, which can reflect the vibration amplitude, frequency, etc. of the paper during printing.

[0046] In operation S220, based on the paper detection model, the paper state in the printing process is determined from the monitoring image data; the text information in the audio information is recognized; the log information is parsed to obtain the initial printing state in the printing process; and the paper vibration curve of the printing process is formed based on the paper vibration information.

[0047] The paper detection model can be a model for detecting paper-related states, which can analyze and process input data related to paper (such as monitoring image data), so as to determine the state of the paper.

[0048] For example, the paper state is various conditions of the paper in the printing process, such as whether the position of the paper is offset, whether paper jam occurs, whether the paper is in a normal conveying state, etc.

[0049] The audio information collected in the printing process is denoised, such as filtering the mechanical noise of the printer, the background noise of paper friction, etc., while the sampling rate and format of the audio information are unified to avoid interference with subsequent recognition. If the segment of the audio information is too long, the audio information can be segmented into short segments to improve processing efficiency. With the aid of a voice recognition model, the extracted information is combined with common texts in the printing scenario (such as printer error prompts “paper jam” “low paper”, status broadcasts “printing starts” “printing completes”, etc.), the recognition range is narrowed, and the accuracy is improved. The corresponding text information is generated according to the matching result.

[0050] The log information is subjected to keyword matching, and the related information of printing is determined according to the printing test requirements. The related information matched can be parsed to obtain the initial printing state of the printing process.

[0051] For example, the relevant information can include: print task basic information (extracting task initiation time, file name, set print quantity, etc. from the log), device state information (screening out toner or ink remaining amount, paper tray paper quantity, whether in online state, etc. record), abnormal prompt information (locating fault related content such as "paper jam error code", "printhead clogging warning").

[0052] For example, the initial printing state can be the state condition at the beginning of the printing process, such as the initial toner remaining amount of the printer, the initial placement position of the paper, etc.

[0053] The paper vibration curve is a visual presentation of the paper vibration information in the form of a curve, which is formed by processing and drawing the related data of the paper vibration (such as the vibration amplitude at different time points), and can clearly show the change of the paper vibration in the printing process.

[0054] In operation S230, based on the preset multi-modal model, the paper state, the text information, the initial printing state and the paper vibration curve are fused to obtain a target fusion feature.

[0055] The preset multi-modal model can be a model that is pre-constructed and trained, which can process multiple different types of data (i.e. multi-modal data), and can integrate and process different types of features such as paper state and text information.

[0056] Fusing the paper state, the text information, the initial printing state and the paper vibration curve can obtain a fusion feature that can reflect the comprehensive information of the printing process.

[0057] In operation S240, according to the target fusion feature, the target printing state of the printing process is detected.

[0058] The target printing state can be the final determination result of the current situation of the printing process.

[0059] For example, the target printing state can be a normal printing state, paper jam or insufficient ink.

[0060] The printing state detection method based on the embodiments of the present application, by acquiring multi-source data such as monitoring image data, log information, audio information and paper vibration information, covers multi-dimensional information such as paper state, initial printing state, text information and paper vibration curve in the printing process. With the help of the preset multi-modal model, the features of these information are fused, which avoids the limitation of single data dimension, can more comprehensively reflect the real state of the printing process, thereby greatly improving the accuracy of the target printing state detection, and reducing the detection deviation caused by the misjudgment of single information.

[0061] In the embodiment of the present application, the text information in the audio information is identified, including: when the noise of the audio information is higher than a preset noise threshold, performing noise reduction processing on the audio information by using a spectral subtraction method; based on a speech recognition model, identifying the noise-reduced audio information to obtain text information.

[0062] First, the spectral subtraction method is used for targeted noise reduction to reduce background noise in the audio information. The noise-reduced audio information can also be standardized to unify the audio format of the audio information, etc., so that the audio data is more suitable for the speech recognition model.

[0063] For example, the background noise can be the mechanical running sound when the printer is running or the friction sound of the paper in the conveying process with the device components, etc.

[0064] Based on the printing state detection method of the embodiment of the present application, the spectral subtraction method is used for targeted noise reduction, which can reduce the interference of the background noise such as the mechanical running of the printer on the effective audio, and greatly reduce the recognition error. By processing the audio information in different noise environments through noise reduction, the situation that the text information cannot be recognized due to environmental or device noise is avoided, so that the identification process can also play a stable role in complex printing scenarios.

[0065] Figure 3 A flowchart for feature fusion according to an embodiment of the present application is schematically shown.

[0066] As shown in Figure 3 The operation S230 of the embodiment of the present application can also include operations S231-S235, and the preset multi-modal model includes a convolution layer, an image encoder, a Transformer layer, a modal alignment layer, a multi-modal fusion layer, and a fusion discrimination layer.

[0067] In operation S231, the convolution layer is used to perform convolution processing on the paper state to obtain paper features, and the paper features include paper edge contour and paper flatness.

[0068] The convolution layer is used for sliding scanning and convolution operation on input data, effectively extracting local key features in the data, and can extract features from related data of the paper state. The paper edge contour can reflect whether the paper is aligned, whether the edge is damaged, etc., which is an important basis for judging the paper placement state; the paper flatness can reflect whether the paper has wrinkles, curling, etc.

[0069] In operation S232, the image encoder is used to encode the paper features to obtain a paper encoding vector.

[0070] The image encoder is usually composed of neural network layers (such as fully connected layers, pooling layers, etc.), which can compress, convert and abstract the input image class features, and map them into a vector form that is easier for subsequent processing by the model. The encoding can refine and integrate the information of the paper features, remove redundant information, and convert it into a fixed-dimensional vector form.

[0071] In operation S233, the text information and the initial print state are sorted respectively using the Transformer layer to obtain a text vector and a state vector.

[0072] The Transformer layer can effectively capture long-distance dependencies in sequence data. When processing sequence information such as text information and initial print state, it adjusts the sequence order in combination with the association between information, and performs operations such as sorting and feature extraction.

[0073] The text vector can be a fixed-dimensional numerical vector converted from the sorted text elements (such as words), which condenses the core semantic features of the text information, facilitating subsequent model module processing. The state vector can be a numerical vector converted from the sorted initial print state elements (such as device parameters), which contains key feature information of the initial print state.

[0074] In operation S234, the text vector and the state vector are aligned using the modal alignment layer to obtain an aligned vector.

[0075] The modal alignment layer can be a functional layer for processing different modal data, aiming to eliminate the semantic bias and dimension difference between different types of vectors such as text vectors and state vectors, so that vectors belonging to different modalities can be associated in the same semantic space.

[0076] For example, alignment can be to form a corresponding association between the text vector and the state vector in terms of semantics, such as establishing a matching relationship between the text vector related to the "paper jam warning" and the state vector corresponding to the "paper feeding exception" parsed from the log, while unifying the dimensions and other attributes of the two.

[0077] In operation S235, the paper encoding vector, the aligned vector and the paper vibration curve are fused by the multi-modal fusion layer to obtain initial fusion features.

[0078] The multi-modal fusion layer can effectively combine feature vector information from different modalities by using specific algorithms (such as attention mechanism, weighted splicing, etc.) to determine the association between modal data. The initial fusion features can be the direct output result of feature fusion, which integrates paper state, text information and initial print state, and paper vibration information.

[0079] In operation S236, the initial fusion feature is discriminated by using the fusion discrimination layer to obtain a target fusion feature, which represents the probability of each of a plurality of preset printing states.

[0080] The fusion discrimination layer can be a functional layer for final processing and discrimination of the initial fusion feature, usually including a classification algorithm or a discrimination model, and can extract key information based on the initial fusion feature and output a discrimination result related to a preset target.

[0081] The preset printing state can be a plurality of typical states preset for judging the printing process, including normal printing, paper jam, insufficient ink, etc., and is an object corresponding to the probability of the target fusion feature.

[0082] The printing state detection method based on the embodiments of the present application can present the target fusion feature in the form of the probability of each preset printing state by analyzing and processing the initial fusion feature, convert the complex multi-modal information in the initial fusion feature into intuitive probability values, clearly reflect the possibility of the printing process being in different states, and provide accurate and quantifiable discrimination basis for subsequent determination of the target printing state, thereby reducing the error of subjective judgment.

[0083] In the embodiments of the present application, the text information includes a plurality of words, and the text information is sorted by using a Transformer layer to obtain a text vector, including: converting the plurality of words into a plurality of word vectors; obtaining a time sequence feature of the word vector, the time sequence feature representing an appearance order of the plurality of words in the printing process; sorting the plurality of word vectors according to the time sequence feature, and determining an abnormal word vector that meets a preset association condition from the plurality of word vectors according to a sorting result; removing the abnormal word vector in the plurality of word vectors, and taking the plurality of word vectors after removal as the text vector.

[0084] The printing state detection method based on the embodiments of the present application can convert the words into word vectors, convert the text information into a numerical form that can be processed by the model, and sort according to the time sequence feature, so as to completely retain the appearance order of the words in the printing process. It can reflect the logical order of the printing related information and avoid semantic deviation caused by disordered order. At the same time, the irrelevant or incorrect abnormal word vector in the text information is removed, so that the printing state can be more accurately detected.

[0085] In the embodiments of the present application, the abnormal word vector that meets the preset association condition is determined from the plurality of word vectors according to the sorting result, including: determining, from the plurality of word vectors, a word vector whose association degree with all adjacent word vectors is less than a preset association threshold according to the sorting result, and taking the word vector as the abnormal word vector.

[0086] Based on the printing state detection method of the embodiment of the present application, the correlation degree between each adjacent word vector is determined through the sorting result, the time sequence logic of information in the printing process can be determined, and the abnormal word vector in the semantic logic which is disconnected with the context before and after in the plurality of word vectors can be accurately locked. After removing the abnormal word vector, the remaining word vector can more completely retain the time sequence semantic chain of the text information, so that the semantics of the final text vector is more coherent and the logic is clearer.

[0087] Figure 4 A flowchart for aligning the text vector and the state vector according to the embodiment of the present application is schematically shown. As shown in Figure 4 , the above operation S234 of aligning the text vector and the state vector of the embodiment of the present application further includes operations S2341-S2345.

[0088] In operation S2341, a semantic matrix is constructed, the semantic matrix takes the plurality of first sub-vectors as rows and takes the plurality of second sub-vectors as columns.

[0089] The number of the first sub-vectors A m is m, the number of the second sub-vectors B n is n, and m and n are both integers greater than 0. The row of the semantic matrix can be [A1, A2, …, Am], and the column of the semantic matrix can be [B1, B2, …, Bn]. m n

[0090] In operation S2342, the cosine similarity algorithm is used to determine the similarity between each first sub-vector and each second sub-vector, and the similarity is filled into the corresponding position of the semantic matrix as an element C ij of the semantic matrix.

[0091] The cosine similarity algorithm is used to calculate the similarity of each first sub-vector and second sub-vector. C ij is the similarity of A i and B j . .

[0092] In operation S2343, elements with a similarity greater than a preset similarity threshold value are selected from the semantic matrix as a plurality of target elements, and the row index and column index of each target element are determined.

[0093] For example, the preset similarity threshold value is 0.7, the similarity of C 12 is 0.8, the similarity of C 21 is 0.6, the similarity of C 33 is 0.9, and the similarity of C 45 is 0.4, then the target elements are C 12 and C 33 . C 12 ​​The row index of C 12 The column index of C 33 The row index of C 12 The column index of C.

[0094] In operation S2344, a target first sub-vector corresponding to the row index is determined from the plurality of first sub-vectors, a target second sub-vector corresponding to the column index is determined from the plurality of second sub-vectors, and the target first sub-vector and the target second sub-vector are combined to form an aligned sub-vector.

[0095] For example, the row index of C 12 The column index of C 12 The first sub-vector corresponding to the row index is A1 and the second sub-vector corresponding to the column index is B2.C 33 The row index of C 33 The column index of C, then the first sub-vector corresponding to the row index is A3 and the second sub-vector corresponding to the column index is B3. The first sub-vector A1 and the second sub-vector B2 form an aligned sub-vector, and the first sub-vector A3 and the second sub-vector B3 form an aligned sub-vector.

[0096] In operation S2345, the weights of the plurality of aligned sub-vectors are determined according to the similarity, and the plurality of aligned sub-vectors are spliced according to the weights to form an aligned vector.

[0097] For example, the similarity of the aligned sub-vector formed by the first sub-vector A1 and the second sub-vector B2 is 0.8, and the similarity of the aligned sub-vector formed by the first sub-vector A3 and the second sub-vector B3 is 0.8. Then the weight of the aligned sub-vector formed by the first sub-vector A1 and the second sub-vector B2 and the aligned sub-vector formed by the first sub-vector A3 and the second sub-vector B3 is 0.8. According to the weight, the two aligned sub-vectors are weighted and spliced to form an aligned vector.

[0098] The printing state detection method based on the embodiment of the present application determines the degree of association between the first sub-vector and the second sub-vector through the semantic matrix and the cosine similarity algorithm, only retains the high-similarity associated pairs, avoids the false matching of irrelevant sub-vectors, accurately aligns the text and the sub-vectors of the state mode, and effectively retains the key associated information between the text vector and the state vector.

[0099] In the embodiment of the present application, the initial fusion feature includes a plurality of sub-features, and the target fusion feature is obtained by using a fusion discrimination layer to discriminate the initial fusion feature, including: determining the association degree of each sub-feature and each preset printing state, the plurality of preset printing states including normal printing, paper jam, and insufficient ink; when the association degree of any target sub-feature in the plurality of sub-features and any target preset printing state in the plurality of preset printing states is higher than a preset association degree threshold, the target sub-feature is classified into the target preset printing state, a feature set of each preset printing state is obtained, and the plurality of feature sets are taken as the target fusion feature.

[0100] For example, each sub-feature corresponds to a specific state information in the printing process, including “paper edge offset 2mm”, “vibration frequency increases by 10Hz”, “audio recognition to ‘low ink’ keyword”, etc.

[0101] The association degree can be a numerical value for measuring the close degree of association between the sub-feature and the preset printing state. The higher the association degree, the more the sub-feature can reflect the characteristics of the corresponding preset printing state. The preset association degree threshold can be a critical value for judging whether the sub-feature can be classified into a certain preset printing state. If the association degree is higher than the critical value, it is considered that the sub-feature is closely associated with the corresponding preset printing state.

[0102] For example, the initial fusion feature includes three sub-features: sub-feature A (“paper conveying stagnation signal”), sub-feature B (“ink cartridge remaining amount display 10%”), and sub-feature C (“printing text clear without broken traces”). The preset printing states are normal printing, paper jam, and insufficient ink, and the preset association degree threshold is 0.6. After calculating the association degree, it is obtained that the association degree of sub-feature A and “paper jam” is 0.8, and sub-feature A is classified into “paper jam” to form a feature set of “paper jam”. The association degree of sub-feature B and “insufficient ink” is 0.7, and sub-feature B is classified into “insufficient ink” to form a feature set of “insufficient ink”. The association degree of sub-feature C and “normal printing” is 0.9, and sub-feature C is classified into “normal printing” to form a feature set of “normal printing”.

[0103] Based on the printing state detection method of the embodiment of the present application, by calculating the association degree of the sub-feature and each preset printing state, and then classifying according to the preset association degree threshold, each sub-feature can be accurately corresponded to the most relevant preset printing state. The mismatch between the sub-feature and the state is avoided, so that the sub-features in each feature set are closely around the corresponding state, and the directivity of the feature to the state is improved.

[0104] In the embodiments of the present application, the target printing state of the printing process is detected according to the target fusion feature, including: determining the probability of a plurality of feature sets according to the number of sub-features in each feature set, the probability representing the ratio of the number of sub-features in each feature set to the number of a plurality of sub-features; determining the preset printing state with the highest probability from a plurality of preset printing states as the target printing state.

[0105] For example, the target fusion feature includes 100 sub-features, the feature set A includes 20 sub-features, the feature set B includes 70 sub-features, and the feature set C includes 10 sub-features. The probability of the preset printing state (paper jam) corresponding to the feature set A is 0.2; the probability of the preset printing state (ink shortage) corresponding to the feature set B is 0.7; and the probability of the preset printing state (normal printing) corresponding to the feature set C is 0.1. It is determined that the target printing state is the preset printing state (ink shortage) corresponding to the feature set B.

[0106] Based on the printing state detection method of the embodiments of the present application, the preset printing state with the highest probability can be directly locked as the target printing state by comparing the probabilities, the result is clear, and the efficiency of printing state detection is improved.

[0107] Based on the above printing state detection method, the present application further provides a printing state detection device. The following will be described in detail Figure 5 with reference to the device.

[0108] Figure 5 The structure block diagram of the printing state detection device according to the embodiments of the present application is schematically shown.

[0109] As Figure 5 shown, the printing state detection device 500 of the embodiments includes a data acquisition module 510, a data processing module 520, a feature fusion module 530, and a state detection module 540.

[0110] The data acquisition module 510 is configured to acquire monitoring image data, log information, audio information, and paper vibration information generated in the printing process. In an embodiment, the data acquisition module 510 can be configured to perform the operation S210 described above, and details are not repeated here.

[0111] The data processing module 520 is configured to determine the paper state in the printing process from the monitoring image data based on a paper detection model; identify the text information in the audio information; parse the log information to obtain the initial printing state in the printing process; and form a paper vibration curve of the printing process based on the paper vibration information. In an embodiment, the data processing module 520 can be configured to perform the operation S220 described above, and details are not repeated here.

[0112] The feature fusion module 530 is configured to perform feature fusion on the paper state, the text information, the initial printing state and the paper vibration curve based on a preset multi-modal model to obtain target fusion features. In an embodiment, the feature fusion module 530 can be configured to perform the operation S230 described above, and details are not described herein again.

[0113] The state detection module 540 is configured to detect a target printing state of the printing process according to the target fusion features. In an embodiment, the state detection module 540 can be configured to perform the operation S240 described above, and details are not described herein again.

[0114] According to an embodiment of the present application, the data processing module 520 includes a noise reduction unit and a recognition unit.

[0115] The noise reduction unit is configured to perform noise reduction processing on the audio information by using a spectral subtraction method when noise of the audio information is higher than a preset noise threshold.

[0116] The recognition unit is configured to recognize the noise-reduced audio information based on a speech recognition model to obtain the text information.

[0117] According to an embodiment of the present application, the feature fusion module 530 further includes a convolution unit, an encoding unit, an ordering unit, an alignment unit, a fusion unit and a discrimination unit.

[0118] The convolution unit is configured to perform convolution processing on the paper state by using a convolution layer to obtain paper features, the paper features including a paper edge contour and a paper flatness.

[0119] The encoding unit is configured to encode the paper features by using an image encoder to obtain a paper encoding vector.

[0120] The ordering unit is configured to perform ordering on the text information and the initial printing state respectively by using a Transformer layer to obtain a text vector and a state vector.

[0121] The alignment unit is configured to perform alignment on the text vector and the state vector by using a modal alignment layer to obtain an alignment vector.

[0122] The fusion unit is configured to perform feature fusion on the paper encoding vector, the alignment vector and the paper vibration curve by using a multi-modal fusion layer to obtain initial fusion features.

[0123] The discrimination unit is configured to perform discrimination on the initial fusion features by using a fusion discrimination layer to obtain target fusion features, the target fusion features representing probabilities of each of a plurality of preset printing states.

[0124] According to an embodiment of the present application, the ordering unit further includes a timing subunit, an ordering subunit and a removal subunit.

[0125] The timing subunit is configured to convert a plurality of words into a plurality of word vectors, and obtain a timing feature of the word vectors, the timing feature representing an appearance order of the plurality of words in a printing process.

[0126] The sorting subunit is configured to sort the plurality of word vectors according to the timing feature, and determine an abnormal word vector that meets a preset association condition from the plurality of word vectors according to a sorting result.

[0127] The removing subunit is configured to remove the abnormal word vector from the plurality of word vectors, and take the plurality of word vectors after the removal as a text vector.

[0128] According to an embodiment of the present application, the sorting subunit is further configured to determine, from the plurality of word vectors, any word vector that has an association degree between any adjacent word vector smaller than a preset association threshold, as the abnormal word vector.

[0129] According to an embodiment of the present application, the alignment unit includes a matrix constructing subunit, an element determining subunit, a vector composing subunit, and a vector determining subunit.

[0130] The matrix constructing subunit is configured to construct a semantic matrix, the semantic matrix taking a plurality of first sub-vectors as rows and a plurality of second sub-vectors as columns.

[0131] The element determining subunit is configured to determine a similarity between each first sub-vector and each second sub-vector by using a cosine similarity algorithm, and fill the similarity into a corresponding position of the semantic matrix as an element of the semantic matrix.

[0132] The vector composing subunit is configured to filter out, from the semantic matrix, elements with a similarity greater than a preset similarity threshold as a plurality of target elements, determine a row index and a column index of each target element, determine a target first sub-vector corresponding to the row index from the plurality of first sub-vectors, determine a target second sub-vector corresponding to the column index from the plurality of second sub-vectors, and compose the target first sub-vector and the target second sub-vector into an aligned sub-vector.

[0133] The vector determining subunit is configured to determine a weight of the plurality of aligned sub-vectors according to the similarity, and splice the plurality of aligned sub-vectors according to the weight to form an aligned vector.

[0134] According to an embodiment of the present application, the discrimination unit includes an association degree determining subunit and a feature determining subunit.

[0135] The association degree determining subunit is configured to determine an association degree of each sub-feature and each preset printing state, the plurality of preset printing states including normal printing, paper jamming, and insufficient ink.

[0136] The feature determination subunit is configured to, when an association degree between any target sub-feature in the plurality of sub-features and any target preset printing state in the plurality of preset printing states is higher than a preset association degree threshold, classify the target sub-feature into the target preset printing state, and obtain a feature set of each preset printing state, and take the plurality of feature sets as target fusion features.

[0137] According to an embodiment of the present application, the state detection module 540 comprises a probability determination unit and a state determination unit.

[0138] The probability determination unit is configured to determine probabilities of the plurality of feature sets according to the number of sub-features in the feature set, and the probability represents a ratio of the number of sub-features in each feature set to the number of the plurality of sub-features.

[0139] The state determination unit is configured to determine a preset printing state with the highest probability in the plurality of preset printing states as a target printing state.

[0140] According to an embodiment of the present application, any plurality of modules in the data acquisition module 510, the data processing module 520, the feature fusion module 530 and the state detection module 540 can be combined in one module for implementation, or any one of the modules can be split into a plurality of modules. Alternatively, at least part of the function of one or more of the modules can be combined with at least part of the function of other modules, and implemented in one module. According to an embodiment of the present application, at least one of the data acquisition module 510, the data processing module 520, the feature fusion module 530 and the state detection module 540 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on board, a system on package, an application specific integrated circuit (ASIC), or any other reasonable manner of hardware or firmware that can be integrated or packaged, or implemented in any one of software, hardware and firmware or in an appropriate combination of any of them. Alternatively, at least one of the data acquisition module 510, the data processing module 520, the feature fusion module 530 and the state detection module 540 can be at least partially implemented as a computer program module that can perform corresponding functions when the computer program module is run.

[0141] Figure 6 A block diagram of an electronic device suitable for implementing the printing state detection method according to an embodiment of the present application is schematically shown.

[0142] As Figure 6As shown, the electronic device 600 according to embodiments of the present application includes a processor 601 which can perform various appropriate actions and processes in accordance with a program stored in a read only memory (ROM) 602 or a program loaded into a random access memory (RAM) 603 from a storage section 608. The processor 601 can include, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor, and / or a related chip set, and / or a dedicated microprocessor (e.g., an application specific integrated circuit (ASIC)), and so on. The processor 601 can also include an on-board memory for cache use. The processor 601 can include a single processing unit or multiple processing units for executing different actions of the method processes according to embodiments of the present application.

[0143] In the RAM 603, various programs and data required for the operation of the electronic device 600 are stored. The processor 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. The processor 601 performs various operations of the method processes according to embodiments of the present application by executing the programs in the ROM 602 and / or the RAM 603. Note that the programs can also be stored in one or more memories other than the ROM 602 and the RAM 603. The processor 601 can also perform various operations of the method processes according to embodiments of the present application by executing the programs stored in the one or more memories.

[0144] According to embodiments of the present application, the electronic device 600 can also include an input / output (I / O) interface 605 which is also connected to the bus 604. The electronic device 600 can also include one or more of the following components connected to the input / output (I / O) interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the input / output (I / O) interface 605 as necessary. A removable medium 611 such as a magnetic disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 610 as necessary, so that a computer program read out therefrom is installed in the storage section 608 as necessary.

[0145] The application further provides a computer readable storage medium, which can be included in the device / apparatus / system described in the above embodiments, or can exist independently without being assembled into the device / apparatus / system. The computer readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the application.

[0146] According to the embodiments of the application, the computer readable storage medium can be a non-volatile computer readable storage medium, which can include, but is not limited to, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any appropriate combination thereof. In this application, a computer readable storage medium can be any tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. For example, according to the embodiments of the application, the computer readable storage medium can include one or more of the above-described ROM 602 and / or RAM 603 and / or one or more memories other than the ROM 602 and the RAM 603.

[0147] The embodiments of the application also include a computer program product, which includes a computer program containing program codes for executing the method shown in the flow chart. When the computer program product is run in a computer system, the program codes are used to make the computer system implement the printing state detection method provided by the embodiments of the application.

[0148] The above-described functions defined in the system / apparatus of the embodiments of the application are performed when the computer program is executed by the processor 601. According to the embodiments of the application, the above-described system, apparatus, module, unit, etc. can be implemented by computer program modules.

[0149] In one embodiment, the computer program can rely on a tangible storage medium such as an optical storage device, a magnetic storage device, etc. In another embodiment, the computer program can also be transmitted, distributed, and downloaded in the form of a signal on a network medium, and be downloaded and installed through the communication part 609, and / or installed from the detachable medium 611. The program codes contained in the computer program can be transmitted by any appropriate network medium, including but not limited to wireless, wired, etc., or any appropriate combination thereof.

[0150] In such embodiments, the computer program can be downloaded and installed from the network via the communication section 609, and / or installed from the removable media 611. When the computer program is executed by the processor 601, the above-described functions defined in the system of the embodiments of the present application are executed. According to the embodiments of the present application, the system, device, apparatus, module, unit, and the like described above can be realized by the computer program modules.

[0151] According to the embodiments of the present application, the program code for executing the computer program provided by the embodiments of the present application can be written in any combination of one or more programming languages, and specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming language, and / or assembly / machine language. The programming language includes, but is not limited to, such as Java, C++, python, "C" language, or similar programming language. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case involving a remote computing device, the remote computing device can be connected to the user computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, connected to the Internet through an Internet service provider).

[0152] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectural, functional, and operational scenarios of systems, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, a program segment, or a portion of code that contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in a different order than that shown in the figures. For example, two blocks that are shown in succession can actually be executed substantially concurrently, or they can sometimes be executed in reverse order, depending on the functionality involved. It should also be noted that each block in the flowcharts or block diagrams, and combinations of blocks in the flowcharts or block diagrams, can be implemented by dedicated hardware-based systems that perform the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0153] Those skilled in the art can understand that the features described in various embodiments of the present application can be combined and / or integrated in various combinations, even if such combinations are not explicitly described in the present application. In particular, the features described in various embodiments of the present application can be combined and / or integrated in various combinations without departing from the spirit and teachings of the present application. All such combinations and / or integrations are within the scope of the present application.

Claims

1. A print state detection method characterized by, The method comprises: acquiring monitoring image data, log information, audio information and paper vibration information generated by a printing process; determining a paper state in the printing process from the monitoring image data based on a paper detection model; recognizing text information in the audio information; parsing the log information to obtain an initial printing state in the printing process; forming a paper vibration curve of the printing process based on the paper vibration information; based on a preset multi-modal model, performing feature fusion on the paper state, the text information, the initial printing state and the paper vibration curve to obtain target fusion features; detecting a target printing state of the printing process according to the target fusion features.

2. The method of claim 1, wherein, The method comprises: when noise of the audio information is higher than a preset noise threshold, performing noise reduction processing on the audio information by using a spectral subtraction method; based on a speech recognition model, recognizing the noise-reduced audio information to obtain the text information.

3. The method of claim 1, wherein, The multi-modal model comprises a convolution layer, an image encoder, a Transformer layer, a modal alignment layer, a multi-modal fusion layer and a fusion discrimination layer; performing feature fusion on the paper state, the text information, the initial printing state and the paper vibration curve to obtain target fusion features comprises: performing convolution processing on the paper state by using the convolution layer to obtain paper features, the paper features comprising a paper edge contour and a paper flatness; performing encoding on the paper features by using the image encoder to obtain a paper encoding vector; performing sorting on the text information and the initial printing state respectively by using the Transformer layer to obtain a text vector and a state vector; aligning the text vector and the state vector by using the modal alignment layer to obtain an alignment vector; performing feature fusion on the paper encoding vector, the alignment vector and the paper vibration curve by using a multi-modal fusion layer to obtain initial fusion features; performing discrimination on the initial fusion features by using the fusion discrimination layer to obtain target fusion features, the target fusion features representing a probability of each of a plurality of preset printing states.

4. The method of claim 3, wherein, The text information comprises a plurality of words, and the method comprises: converting the plurality of words into a plurality of word vectors; obtaining time sequence features of the word vectors, the time sequence features representing an appearance order of the plurality of words in the printing process; sorting the plurality of word vectors according to the time sequence features, and determining an abnormal word vector that satisfies a preset association condition from the plurality of word vectors according to a sorting result; removing the abnormal word vector from the plurality of word vectors, and taking the plurality of word vectors after removal as a text vector.

5. The method of claim 4, wherein, The method comprises: from the plurality of word vectors, determining, according to the sorting result, a word vector whose association degree with all adjacent word vectors is less than a preset association threshold, and taking the word vector as the abnormal word vector.

6. The method of claim 3, wherein, The text vector includes a plurality of first sub-vectors, and the state vector includes a plurality of second sub-vectors; the modal alignment layer is used to align the text vector and the state vector to obtain an aligned vector, including: A semantic matrix is constructed, the semantic matrix taking the plurality of first sub-vectors as rows and the plurality of second sub-vectors as columns; A cosine similarity algorithm is used to determine the similarity between each first sub-vector and each second sub-vector, and the similarity is filled into the corresponding position of the semantic matrix as an element of the semantic matrix; Elements with a similarity greater than a preset similarity threshold are screened out from the semantic matrix as a plurality of target elements, and the row index and the column index where each target element is located are determined; A target first sub-vector corresponding to the row index is determined from the plurality of first sub-vectors, and a target second sub-vector corresponding to the column index is determined from the plurality of second sub-vectors, and the target first sub-vector and the target second sub-vector are combined to form an aligned sub-vector; According to the similarity, the weights of a plurality of aligned sub-vectors are determined; According to the weights, a plurality of aligned sub-vectors are spliced to form the aligned vector.

7. The method of claim 3, wherein, The initial fusion feature includes a plurality of sub-features, and the fusion discriminant layer is used to discriminate the initial fusion feature to obtain a target fusion feature, including: The association degree of each sub-feature with each preset printing state is determined, and the plurality of preset printing states include normal printing, paper jamming, and insufficient ink; When the association degree of any target sub-feature in the plurality of sub-features with any target preset printing state in the plurality of preset printing states is higher than a preset association degree threshold, the target sub-feature is classified into the target preset printing state, and a feature set of each preset printing state is obtained, and a plurality of feature sets are taken as target fusion features.

8. The method of claim 7, wherein, According to the target fusion feature, the target printing state of the printing process is detected, including: According to the number of sub-features in the feature set, the probability of a plurality of feature sets is determined, and the probability represents the ratio of the number of sub-features in each feature set to the number of sub-features; The preset printing state with the highest probability in the plurality of preset printing states is determined and taken as the target printing state.

9. A print state detection apparatus characterized by comprising: The device includes: A data acquisition module is configured to acquire monitoring image data, log information, audio information, and paper vibration information generated by a printing process; A data processing module is configured to determine a paper state in the printing process from the monitoring image data based on a paper detection model; identify text information in the audio information; parse the log information to obtain an initial printing state in the printing process; and form a paper vibration curve of the printing process based on the paper vibration information; A feature fusion module is configured to perform feature fusion on the paper state, the text information, the initial printing state, and the paper vibration curve based on a preset multi-modal model to obtain a target fusion feature; and A state detection module is configured to detect a target printing state of the printing process according to the target fusion feature.

10. An electronic device, comprising: one or more processors; memory storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method of any one of claims 1-8.

11. A computer readable storage medium having stored thereon a computer program or instructions, characterized in that, The computer program or instructions, when executed by a processor, implement the steps of the method of any one of claims 1-8.

12. A computer program product comprising computer programs or instructions, characterized in that, The computer program or instructions, when executed by a processor, implement the steps of the method of any one of claims 1-8.

Images