Industrial data identification method, device, equipment, medium and product

Abstract

The invention discloses an industrial data identification method and device, equipment, a medium and a product, and relates to the technical field of data identification, and the method comprises the steps: obtaining a display image of industrial equipment; preprocessing the display image to obtain a preprocessed image; performing image segmentation processing on the preprocessed image to obtain a sub-image, and determining a target area containing character information from the sub-image; and identifying the target area to obtain text data, and correcting the text data to generate industrial data corresponding to the industrial equipment. According to the method, a data acquisition gateway does not need to be additionally arranged on the industrial equipment, the problem that the operation efficiency of the equipment is influenced is avoided, the text data of the industrial equipment can be accurately identified, identification errors are effectively reduced, and the identification accuracy of the industrial data is further improved.

Description

Technical Field

[0001] The present application relates to the technical field of data recognition, and particularly to an industrial data recognition method, device, equipment, medium and product. Background Art

[0002] With the rapid development of industrial informatization, a large amount of data is generated in the industrial production process. These data come from a wide range of sources, including operation parameter records of various industrial equipment, production reports, quality inspection reports, product labels, etc. Among them, industrial data, as a key indicator for enterprise management, plays a crucial role in enterprise production management, quality control, equipment maintenance, and decision-making. For example, it can quickly identify and respond to abnormal situations on the production line, reduce downtime and fault repair cycles, and improve production continuity and stability. Therefore, in order to enhance the competitiveness of industrial enterprises and achieve intelligent transformation, it is particularly important to efficiently and accurately identify industrial data.

[0003] Currently, in the related art, data is collected and recognized through the communication protocol inside industrial equipment. However, this method requires relying on a large number of communication interfaces and data definitions provided by equipment manufacturers. After purchasing the equipment, enterprises need to spend a large amount of cost to purchase communication authorizations. Moreover, for some old equipment, due to the weak processing ability of the control system, the operation efficiency of the equipment is affected after installing the data acquisition gateway, resulting in low accuracy of industrial data recognition. Summary of the Invention

[0004] The purpose of the present application is to provide an industrial data recognition method, device, equipment, medium and product.

[0005] To achieve the above purpose, the present application provides the following solutions: In the first aspect, the present application provides an industrial data recognition method, including: Obtaining a display image of an industrial equipment; the display image is obtained by shooting or screenshot; Performing preprocessing on the display image to obtain a preprocessed image; Performing image segmentation processing on the preprocessed image to obtain sub-images, and determining a target area containing character information from the sub-images; Recognizing the target area to obtain text data, and performing correction processing on the text data to generate industrial data corresponding to the industrial equipment.

[0006] In the second aspect, the present application provides an industrial data recognition device, including: An obtaining module, configured to obtain a display image of an industrial equipment; the display image is obtained by shooting or screenshot; A preprocessing module, configured to perform preprocessing on the display image to obtain a preprocessed image; A segmentation module, configured to perform image segmentation processing on the preprocessed image to obtain sub-images, and determine a target area containing character information from the sub-images; A correction module, configured to identify the target area to obtain text data, and perform correction processing on the text data to generate industrial data corresponding to the industrial device.

[0007] In a third aspect, the present application provides a computer device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor executes the computer program to implement the steps of the industrial data recognition method described in any one of the above.

[0008] In a fourth aspect, the present application provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the industrial data recognition method described in any one of the above are implemented.

[0009] In a fifth aspect, the present application provides a computer program product, including a computer program, and when the computer program is executed by a processor, the steps of the industrial data recognition method described in any one of the above are implemented.

[0010] According to the specific embodiments provided by the present application, the following technical effects are disclosed in the present application: The technical solution provided by the present application can automatically obtain the display image of the industrial device, perform preprocessing on the display image to obtain the preprocessed image, thereby providing good data guidance information for subsequent data recognition, and perform image segmentation processing on the preprocessed image, so as to accurately obtain sub-images, enabling a more fine-grained determination of the target area containing character information from each sub-image, then identifying the target area to obtain text data, and further performing correction processing on the text data, effectively reducing recognition errors and further improving the accuracy of industrial data recognition. Description of the Drawings

[0011] Figure 1 It is an application environment diagram of an industrial data recognition method provided by an embodiment of the present application; Figure 2 It is a flow schematic diagram of an industrial data recognition method provided by an embodiment of the present application; Figure 3 It is a structural schematic diagram of processing the preprocessed image to obtain the target area provided by an embodiment of the present application; Figure 4 It is a method flow schematic diagram of performing a screenshot operation on the target page of the industrial device provided by an embodiment of the present application; Figure 5Schematic diagram of switching from page A to page B provided by an embodiment of the present application; Figure 6 Schematic diagram of switching from page A to page B and page C provided by an embodiment of the present application; Figure 7 Schematic diagram of switching from page A and page B to page C provided by an embodiment of the present application; Figure 8 Schematic diagram of switching from page A and page B to page C and page D provided by an embodiment of the present application; Figure 9 Schematic flowchart of a method for constructing an image segmentation model provided by an embodiment of the present application; Figure 10 Schematic diagram of processing the text to be corrected into corresponding industrial data provided by an embodiment of the present application; Figure 11 Schematic diagram of functional modules of an industrial data recognition device provided by an embodiment of the present application; Figure 12 Schematic diagram of the structure of a computer device provided by an embodiment of the present application. Detailed implementation manners

[0012] The present application will be further described in detail below with reference to the accompanying drawings and specific implementation manners.

[0013] The industrial data recognition method provided by the embodiments of the present application can be exemplarily applied to an application environment as shown in Figure 1 Among them, the application environment may at least include industrial equipment, and may further include at least one of a data storage system and a server 104.

[0014] A display area is provided on the above-mentioned industrial equipment, and the display area is used to display various information of the industrial equipment, such as equipment operation status information, production process information, and operation guidance information. The display area may include one or more.

[0015] The industrial equipment 102 can communicate with the server 104 through a network.

[0016] The data storage system can store data related to the present application, such as one or more of display images, industrial data, historical texts, and preset standard texts (historical texts and preset standard texts will be introduced later). Figure 1 The data storage system in can be set separately, integrated on the server 104, or placed on the cloud or other servers.

[0017] In some embodiments, the above application environment may further include an image acquisition device. The image acquisition device can acquire the display image of the industrial device and send it to the industrial device 102. Then, the industrial device 102 can send the acquired display image to the server 104 (or the image acquisition device can also send the display image to the server 104 by itself). After receiving the display image, the server 104 performs preprocessing, segmentation, recognition, and correction processing on the display image to obtain the industrial data corresponding to the industrial device. The server 104 can feedback the obtained industrial data to the industrial device 102 to visualize the industrial data through the industrial device 102. In addition, in the case of the existence of a data storage system, the server 104 can also store the industrial data in the data storage system.

[0018] In some other embodiments, the above application environment may further include an image processing device. The image processing device can receive the above display image and perform preprocessing, segmentation, recognition, and correction processing on the display image to obtain the industrial data corresponding to the industrial device. Further, the image processing device can also perform one or more of the following steps: uploading the industrial data to the server 104 and / or the data storage system; feedbacking the industrial data to the industrial device 102.

[0019] In addition, in some embodiments, the above industrial data recognition method can also be implemented by the industrial device 102 alone. For example, the industrial control program inside the industrial device 102 can acquire the display image of the industrial device and perform preprocessing, segmentation, recognition, and correction processing on the acquired display image to obtain the industrial data corresponding to the industrial device. Further, the industrial device 102 can also perform the following steps: uploading the industrial data to the server 104 / data storage system.

[0020] Alternatively, in some embodiments, the above recognition method can be completed by the cooperation of the industrial device 102 and the server 104: the industrial control program inside the industrial device acquires the display image of the industrial device and uploads it to the server 104, and the server 104 performs preprocessing, segmentation, recognition, and correction processing on the display image to obtain the industrial data corresponding to the industrial device. In addition, the server 104 can also perform one or more of the following steps: storing the industrial data in the data storage system; feedbacking the industrial data to the industrial device 102.

[0021] In one example, the types of the above industrial devices may include: production and manufacturing types, detection and monitoring types, energy and power types, etc. Exemplary production and manufacturing type industrial devices may include: industrial control all-in-one machines; Exemplary detection and monitoring type industrial devices may include: intelligent detection devices, environmental detection devices, etc.; Exemplary energy and power type industrial devices may include: power monitoring system devices, smart meters, etc.

[0022] Exemplarily, the above-mentioned server 104 can be a server of a database, a central control system, or a data center. In addition, the server 104 can be implemented by an independent server or a server cluster composed of multiple servers, and can also be a cloud server, etc.

[0023] In other application scenarios, the above-mentioned server 104 can also be replaced by a host computer or an edge computing device, etc., or the server 104 coexists with the host computer / edge computing device.

[0024] Next, based on the common aspects involved in the present invention above, the embodiments of the present invention will be further described in detail.

[0025] In an exemplary embodiment, as Figure 2 shown, an industrial data recognition method is provided. Exemplarily, this method can be executed independently by an industrial device, or jointly executed by an industrial device and a server / host computer / edge computing device, etc., or jointly executed by an industrial device and an image processing device, etc.

[0026] In the embodiments of the present application, the following steps S201 to step S204 are included. Among them: Step S201, obtain the display image of the industrial device.

[0027] Exemplarily, the display image can be obtained by photographing or taking a screenshot of the target page, or by photographing the display area of the industrial device.

[0028] In one example, those skilled in the art can select a photographing or screenshot method to obtain the display image according to the device type of the industrial device.

[0029] The device types of the above-mentioned industrial devices can include, for example, single-mode display industrial devices and multi-mode display industrial devices. When the device type of the industrial device is a multi-mode display industrial device, the screenshot method can be selected to collect its display image. Specifically, the industrial control program inside the industrial device can perform a screenshot operation on the target page of the industrial device to obtain the display image; when the device type of the industrial device is a single-mode display industrial device, the industrial camera can be selected to perform an image acquisition operation on the display area of the industrial device to obtain the display image. Of course, the photographing method can also be selected to obtain the image of the target page of the multi-mode display industrial device, or (when the single-mode display industrial device supports screenshot) the screenshot method can be selected to obtain the display image of the single-mode display industrial device, which will not be elaborated here.

[0030] Single-mode industrial equipment refers to industrial equipment with a single mode or function. For example, single-mode industrial equipment can be industrial instruments, sensors, industrial monitoring equipment, etc. It has a single display screen, and its display content and mode are relatively fixed. It does not have multiple display functions or the ability to switch modes. It only displays simple text information or status indicators to provide basic equipment operation status. Among them, the sensor can be a temperature sensor, a pressure sensor, a humidity sensor, etc.

[0031] Multi-mode industrial equipment refers to industrial equipment with multiple display modes or functions. The multi-mode industrial equipment has greater flexibility and adaptability, and can switch or simultaneously present multiple types of images, data or information according to different industrial application requirements. For example, multi-mode industrial equipment may include industrial computers or touch screen displays, which can display conventional text and graphic information for equipment operation interface and parameter setting; can also switch to image or video display mode for monitoring video images of production sites; and may also support the display of complex engineering drawings, flow charts and other files in multiple formats. In addition, some multi-mode industrial equipment may also have different display resolutions, color modes or viewing angle adjustment functions to meet the requirements of different environments and tasks.

[0032] In one example, step S201 and subsequent steps may be performed when the image acquisition condition is met.

[0033] The image acquisition conditions may include at least one of a scene trigger condition, a state trigger condition and a time trigger condition. The state trigger condition may be, for example, being in a stable operating state. In actual applications, the operating state information of the industrial equipment may be obtained by the monitoring device, and whether it is in a stable operating state may be determined based on the operating state information. Due to the different application scenarios of industrial equipment, the above-mentioned operating state information may also include different parameters, such as one or more of temperature, pressure, flow, humidity, gas concentration, etc. As for the stable operating state, it may refer to the operating state parameters (such as temperature, pressure, flow, etc.) of the industrial equipment not changing within a preset time period, or the change value is less than the set threshold value (the change value here can be obtained by subtracting the minimum value from the maximum value of the operating state parameter within the preset time period); those skilled in the art may flexibly design the value of the preset time period and the value of the threshold value according to actual conditions, which will not be elaborated here.

[0034] The scene trigger condition may be, for example, that the current scene satisfies the preset scene condition; the time trigger condition may be that the duration reaches the preset duration and / or reaches the predetermined time point. For example, the time trigger condition may be to set the preset time point to 12:00, and when the time reaches 12:00, perform the image acquisition operation; for another example, the preset duration may be set to 10 minutes, that is, perform the image acquisition operation every 10 minutes. The above-mentioned preset duration and predetermined time point may be set temporarily, or they may be fixed once set. Those skilled in the art may make flexible designs, which will not be elaborated here.

[0035] As an implementable method, when the industrial equipment is a single-mode display industrial equipment, the corresponding image acquisition conditions may exemplarily include time trigger conditions. In the process of acquiring the display image of the industrial equipment, an industrial camera may be pre-installed, and the installation position should ensure that the field of view of the industrial camera can cover the display area or target area of ​​the industrial equipment (the target area may be a part of the display area) and can capture a clear picture. When it is detected that the time trigger condition is met, the industrial camera is controlled to capture the display area of ​​the single-mode display industrial equipment, thereby acquiring the display image of the industrial equipment.

[0036] As another possible implementation method, if the industrial device is a single-mode display industrial device, it can be designed to execute step 201 when a scene trigger condition is met.

[0037] Among them, the scene triggering condition can be flexibly designed according to the actual application scenario. For example, the smoke concentration greater than the smoke threshold can be designed as a scene triggering condition. In the specific implementation, a third-party device (such as a control device) can send a smoke detection instruction to the smoke sensor, and the smoke sensor receives and responds to the smoke detection instruction, detects the smoke concentration and sends it to the control device (of course, the smoke sensor can also periodically detect the smoke concentration and send it to the control device). When the control device detects that the smoke concentration is greater than the smoke threshold, it sends an image acquisition instruction to the industrial camera, controls the industrial camera to capture the display image and sends the display image to the industrial device, so that the industrial device takes pictures to obtain the display image. Alternatively, when it is detected that the smoke concentration is greater than the smoke threshold, an image acquisition instruction or an alarm indication can be sent to the industrial device, thereby triggering the industrial device to send an image acquisition instruction to the industrial camera.

[0038] As another possible implementation method, when the industrial equipment is a multi-mode industrial equipment, exemplary designable image acquisition conditions include time trigger conditions (in other implementation methods, they may also be state trigger conditions and / or scene trigger conditions). When the time trigger conditions are met, a screenshot operation is performed to obtain the image displayed on the display area of ​​the industrial equipment, so as to subsequently analyze these real images, monitor the equipment operating status, record the work process, and other purposes.

[0039] Step S202: Preprocess the displayed image to obtain the preprocessed image.

[0040] After obtaining the displayed image, preprocess the displayed image, including denoising, grayscale conversion, and contrast enhancement operations, to obtain the preprocessed image.

[0041] It should be noted that during the acquisition of the displayed image, there may be interference from the image acquisition device or the industrial environment, resulting in possible noise in the displayed image. Therefore, during the preprocessing of the displayed image, it is necessary to first remove the noise in the displayed image to obtain the image after noise removal, so as to improve the image quality. Then, perform grayscale conversion on the image after noise removal to obtain a grayscale image, which can reduce the amount of data, facilitate subsequent processing, and can also highlight the brightness information of the image, which is more conducive to feature extraction and analysis in some applications. Then, perform contrast enhancement processing on the grayscale image to obtain the preprocessed image, thereby improving the visual effects such as contrast and brightness of the image, making the details in the image clearer, and facilitating subsequent analysis and processing.

[0042] During the process of removing noise in the displayed image, a filtering algorithm can be used for processing to obtain the image after noise removal. The filtering algorithm can include at least one of mean filtering, median filtering, and Gaussian filtering, etc. Among them, mean filtering takes the average value within the pixel neighborhood to replace the value of the pixel, which can effectively reduce random noise. Median filtering replaces the pixel value with the median value within the neighborhood, which has a good inhibitory effect on salt-and-pepper noise, etc., and can better preserve the image edges. Gaussian filtering performs weighted averaging on the neighborhood pixels according to the Gaussian function, and can preserve the details of the image while removing noise.

[0043] During the process of grayscale conversion of the image after noise removal, the weighted average method can be used. According to the sensitivity of the human eye to different color channels, obtain the weights of the RGB three channels in the image after noise removal, and perform weighted summation on the channel values corresponding to the RGB three channels according to the weights to obtain the corresponding grayscale value, thereby generating a grayscale image.

[0044] During the process of contrast enhancement processing of the grayscale image, an image enhancement algorithm can be used. The image enhancement algorithm is, for example, histogram equalization. By adjusting the histogram of the grayscale image, the grayscale value distribution of the grayscale image is made more uniform, thereby enhancing the contrast of the image. In addition, the image can also be enhanced by adjusting the brightness and contrast parameters of the image. For example, using the linear transformation method I out =a×I in +b, where I in is the pixel value of the input grayscale image, I outis the pixel value of the pre - processed output image, and a and b are parameters for adjusting contrast and brightness, which can be customized.

[0045] Optionally, during the pre - processing of the displayed image, image cropping and scaling, geometric correction, etc. can also be used for pre - processing.

[0046] Step S203: Perform image segmentation on the pre - processed image to obtain sub - images, and determine the target area containing character information from the sub - images.

[0047] Among them, when there are multiple sub - images, the sizes of each sub - image can be the same or different.

[0048] Exemplarily, the pre - processed image, sub - images, and target area can be seen in Figure 3 shown. Exemplarily, the target area can be represented in the form of a frame. The shape of the frame can be, for example, a circular area, a rectangular area, a rhombus area, or a triangular area. In this embodiment, the shape of the frame is not limited in any way, as long as it can contain the character information.

[0049] Among them, the character information in the above - mentioned target area can include at least one of the following: numbers, letters, Chinese characters, special characters, etc.

[0050] Subsequent in this article, a more detailed introduction will be given on how to perform segmentation processing and determine the target area.

[0051] Step S204: Recognize the target area to obtain text data, and perform correction processing on the text data to generate industrial data corresponding to the industrial equipment.

[0052] In one example, after obtaining the target area, an identification algorithm can be used to perform identification processing on the target area. The identification algorithm can be, for example, Optical Character Recognition (OCR), or a neural network model, a template matching algorithm, a support vector machine algorithm, etc. By using the identification algorithm to perform identification processing on the target area, text data can be obtained. The text data can include at least one of the following: numbers, letters, Chinese characters, special characters, etc.

[0053] In an exemplary embodiment, during the process of using the template matching algorithm to recognize the text data in the target area, the data in the target area to be recognized can be compared with a predefined text template, calculate the similarity between the two, and extract the data with a similarity greater than the preset similarity threshold as the text data.

[0054] In another exemplary embodiment, in the process of using a neural network model to identify text data in a target area, the neural network model may include a convolutional neural network (CNN) and a recurrent neural network (RNN). By using the CNN to extract the spatial features of characters in the target area and using the RNN to process the sequential information of the characters, it is possible to identify irregular text and thus obtain the text data.

[0055] Further, after the text data is determined, correction processing can be performed on the text data. For example, the text data can be compared with a reference text (such as the historical text or preset standard text mentioned later), and the text data with inconsistent comparison results is determined as the data to be corrected. Another example is to check whether the text data meets the preset change conditions, and the text data that does not meet the preset change conditions is used as the data to be corrected. Then, correction processing is performed on the data to be corrected to generate the industrial data corresponding to the industrial device. Among them, the preset change conditions can be within a certain change range and can be customized according to requirements.

[0056] Compared with the prior art, the technical solution in the above embodiment does not require installing a data acquisition gateway on the industrial device, avoiding the problem of affecting the device operation efficiency, nor does it need to implement data acquisition through the communication protocol inside the industrial device. When the image acquisition conditions are met, it can automatically obtain the display image of the industrial device, preprocess the display image to obtain the preprocessed image, thereby providing good data guidance information for subsequent data recognition, and perform image segmentation processing on the preprocessed image to accurately obtain sub-images, enabling a more fine-grained determination of the target area containing character information from each sub-image, then identifying the target area to obtain text data, and further performing correction processing on the text data, effectively reducing recognition errors and further improving the accuracy of industrial data recognition.

[0057] The following provides a more detailed introduction to the relevant content of obtaining the display image through screenshots.

[0058] As mentioned above, the display image can be an image obtained by the industrial control program inside the industrial device performing a screenshot operation on the target page of the industrial device.

[0059] The target page is the page to be subjected to data recognition or the page for displaying the parameters of interest, and it may include character information. The target page corresponding to each industrial device can be one, two, or more.

[0060] In one example, the above industrial control program can be a program for controlling each module of industrial equipment, which is usually written using specialized programming languages and development tools, such as languages like LabVIEW, QT, WinForm, VB, etc., and the corresponding programming software. Taking the recognition algorithm as OCR as an example, the above program can be closely configured with each module in the industrial equipment to achieve precise control. It can integrate the OCR acquisition module into the industrial control program of the industrial equipment. Exemplarily, the OCR acquisition module can be used to perform operations such as collecting display images and data recognition to obtain industrial data. That is, the foregoing steps 201-204 can be executed by the OCR acquisition module. In addition, the OCR acquisition module can also collect the display image and output the display image to an external device for processing. That is, the foregoing step 201 can be executed by the OCR acquisition module, and the foregoing steps 202-204 can be executed by the external device. Of course, the foregoing steps 201-202 can also be executed by the OCR acquisition module to output the preprocessed image, and the foregoing steps 203-204 can be executed by the external device. Or, the foregoing steps 201-203 can be executed by the OCR acquisition module, and the foregoing step 204 can be executed by the external device. Those skilled in the art can make flexible designs and will not be elaborated here.

[0061] The screenshot operation can obtain the display image more clearly, avoiding problems such as easy recognition errors in complex industrial environments (such as uneven lighting and reflection on the device surface), reducing the dependence on high-precision equipment and professional technicians, lowering the equipment procurement and maintenance costs, and providing good data guidance information for subsequent data recognition.

[0062] There are various implementation methods for performing the screenshot operation on the target page. Below, taking the industrial control program integrated with the OCR acquisition module (abbreviated as the industrial control program) as an example, two exemplary implementation methods will be introduced: Target page screenshot implementation method 1: The above industrial control program performs a screenshot operation on all display pages of the industrial equipment and filters out the screenshots of non-target pages among all the screenshots.

[0063] The display page is a general term for the target page and the non-target page. For example, assuming there are 100 display pages in total, among which 10 are target pages and 90 are non-target pages, then all 100 display pages can be screenshot, and the screenshots of 90 non-target pages can be filtered out.

[0064] Exemplarily, performing a screenshot on all display pages can include the following steps: Controlling each display page in the display area of the industrial equipment by the industrial control program to perform a screenshot operation in sequence according to the display order or randomly to obtain the display images of all display pages of the industrial equipment. The display page can include non-target pages and target pages.

[0065] When implemented, after taking screenshots of all display pages, all the screenshots can be filtered (abbreviated as "take all screenshots and then filter"), or filtering can be performed once for each screenshot operation (abbreviated as "take screenshots and filter simultaneously").

[0066] There can be multiple filtering methods, and two methods are introduced below.

[0067] Filtering method 1: All display images can be compared with a preset display template, and the display images that do not match the preset display template among all display images are removed, and the display images that match the preset display template among all display images are used as the display images corresponding to the target pages. The preset display template is custom-set in advance according to the display pages required by the user. Filtering method 1 can be applied to "take all screenshots and then filter" or "take screenshots and filter simultaneously".

[0068] In one example, during the process of comparing all display images with the preset display template, it can be to compare the layout features, constituent elements, textures, and colors of each display image with the layout features, constituent elements, textures, and colors in the preset display template one by one, filter the images with inconsistent layout features, constituent elements, textures, and colors in the comparison, and use the images with consistent layout features, constituent elements, textures, and colors as the display images of the industrial equipment.

[0069] Filtering method 2: A unique page identifier (such as a page number) can be set in advance for each display page, and the page identifier of the currently displayed page can be dynamically stored in a certain storage space. After taking a screenshot of the currently displayed page, the page identifier of the currently displayed page can be compared with the page identifiers of all target pages. If it is different from the page identifiers of all target pages, it is discarded, and the image corresponding to the page with the consistent page identifier is used as the display image. Filtering method 2 can be applied to "take screenshots and filter simultaneously".

[0070] It is mentioned above that the image acquisition conditions include at least one of a scene trigger condition, a status trigger condition, and a time trigger condition.

[0071] Taking the image acquisition condition including the time trigger condition as an example, screenshots and filtering can be performed triggered by time.

[0072] Taking the image acquisition conditions including the scene trigger condition and the time trigger condition as an example, after meeting the scene trigger condition, screenshots and filtering can be performed triggered by time, and the methods of taking screenshots and filtering can refer to the foregoing records.

[0073] Taking the image acquisition conditions including the scene trigger and the status trigger conditions as an example, after simultaneously meeting the scene trigger condition and the status trigger condition, all display images can be "taken all screenshots and then filtered" or "taken screenshots and filtered simultaneously".

[0074] Taking the example where the image acquisition conditions simultaneously include scene trigger, time trigger, and status trigger conditions, after simultaneously satisfying the scene trigger condition and the status trigger condition, screenshots and filtering can be performed with time as the trigger. For the methods of taking screenshots and filtering, reference can be made to the foregoing description.

[0075] The following introduces several examples of taking screenshots and filtering with time as the trigger: Taking the example where the duration reaches a preset duration (assumed to be 10 seconds) as the image acquisition condition, every time the preset duration is reached, a display page can be displayed in the display order or randomly, and a screenshot operation can be performed on it. After taking screenshots of all display pages, filtering is performed. Assuming there are 100 display pages in total, then a screenshot is taken every 10 seconds. Figure 1 After taking screenshots of all these 100 pages, filtering is performed; Alternatively, every time the preset duration is reached, a display page can be displayed in the display order or randomly, and a screenshot operation can be performed on it. Then, it is determined whether the captured image is a target page. If so, it is retained; if not, it is discarded. Continuing with the previous example, that is, edge - capturing and filtering are performed every 10 seconds; Alternatively, every time the preset duration is reached, screenshots are taken of all display pages. Figure 1 After that, filtering is performed on all the screenshots, that is, full - capturing and then filtering are performed every 10 seconds; Alternatively, every time the preset duration is reached, screenshots are taken of all display pages. Figure 1 Every time an image is captured, it is determined whether it is a target page. If so, it is retained; if not, it is discarded. Still taking the preset duration of 10 seconds and 100 display pages as an example, then edge - capturing and filtering are performed 100 times every 10 seconds.

[0076] The second implementation method for capturing screenshots of the target page: In this implementation method, a screenshot operation is performed on the target page, and no screenshot operation is performed on non - target pages.

[0077] Please refer to Figure 4 As shown, the second implementation method for capturing screenshots of the target page may include: Step S301, display each target page in the display area of the industrial device.

[0078] One target page can be displayed in the display area each time, or multiple target pages can be displayed simultaneously. When multiple target pages are displayed in the display area, the page size dimensions of each target page can be the same or different.

[0079] Or rather, in each display operation, at least one target page is displayed.

[0080] Step S302: taking a screenshot of (each) target page displayed in the display area.

[0081] By executing S301 and S302 in a loop, a screenshot operation can be performed on each target page.

[0082] It is mentioned above that the image acquisition conditions include at least one of a scene trigger condition, a state trigger condition and a time trigger condition.

[0083] Taking the case where the image acquisition condition includes a time trigger condition as an example, steps 301 and 302 may be executed based on time.

[0084] Taking the image acquisition conditions including the scene trigger condition and the time trigger condition as an example, after the scene trigger condition is met, steps 301 and 302 may be executed with time as a trigger.

[0085] Taking the image acquisition condition including the scene trigger condition and the state trigger condition as an example, steps 301 and 302 may be performed after both the scene trigger condition and the state trigger condition are satisfied.

[0086] Taking the example that the image acquisition conditions include scene trigger, time trigger and state trigger conditions at the same time, after the scene trigger condition and the state trigger condition are simultaneously satisfied, steps 301 and 302 may be executed with time as a trigger.

[0087] Several examples of executing steps 301 and 302 using time as a trigger are described below: Taking the image acquisition condition that the duration reaches a preset duration (assuming 10 seconds) as an example, each time the preset duration is reached, step 301 is performed once to display at least one target page, and then step 302 is performed once to take a screenshot of the displayed at least one target page. Similarly, each time the preset duration is reached, steps 301 and 302 are performed once.

[0088] Alternatively, each time a preset time period is reached, steps 301 and 302 may be executed in a loop until all target pages are captured.

[0089] The following description focuses on the implementation of step 301.

[0090] In other embodiments of the present application, the above step 301 may exemplarily include: obtaining a page switching configuration file; and displaying a target page according to the page switching configuration file.

[0091] It should be noted that the above page switching configuration file refers to the file corresponding to page switching, which can be custom - set by the user in advance according to actual needs. The page switching configuration file can configure each page, or can configure each target page. That is to say, the page switching file can be configured for each display page respectively, or the page switching file can be configured for each target page respectively. It is also possible for all display pages or all target pages to share a page switching configuration file, or for some display pages or target pages to share the page switching configuration file. Those skilled in the art can design flexibly and will not be elaborated here.

[0092] Exemplarily, the page switching configuration file at least includes: page switching control position information.

[0093] At least one interaction operation on the page switching control can be performed according to the page switching control position information to switch from the current page to the next target page.

[0094] The interaction operation can include at least one of click operation, input operation, etc.

[0095] The above - mentioned page switching control position information is used to represent the position of the page switching control. The click, input, etc. operations of the user can be simulated using the page switching control position information to achieve automatic click and input. In one example, the page switching control position information (abbreviated as position information) can include the coordinates of a preset point in the page switching control (such as the abscissa and ordinate). Exemplary preset point coordinates can include the center point coordinates, the upper - left corner coordinates (if the control is rectangular), etc. In another example, if the control is rectangular, the position information can also include its upper - left corner coordinates, as well as the length and width of the rectangle, and the position of the page switching control can also be determined.

[0096] The page switching control can be set in the page of the industrial device. Of course, it can also exist independently of the page. For example, it can be located in the display area of the industrial device in a floating manner, etc. The page switching control can include one or more of a switching input box, a switching button, and a confirmation button.

[0097] In other examples, the page switching configuration file can also include at least one of a page switching path and target page indication information.

[0098] Next, how to directly switch to the next target page based on the switching input box will be introduced.

[0099] In one embodiment, a corresponding page switching configuration file is set for each display page, which can include the position information of the switching input box in the corresponding display page, and the page switching path of the next target page. Then step 301 can include the following steps: Based on the location information, input the page switching path of the next target page into the switching input box of the currently displayed page (this step is used to simulate keyboard input. In addition, the number of next target pages can be multiple).

[0100] According to the input page switching path, switch the page displayed on the industrial device from the current page to the next target page.

[0101] In this embodiment, the page switching path can be the storage address of the next target page; when there are multiple next target pages, at least one storage address can be input into the switching input box (if there are multiple storage addresses, they can be separated by a delimiter). The switching input box obtains the target page according to the storage addresses of each next target page and displays it on the display screen of the industrial device at the same time.

[0102] In addition, in other embodiments of the present application, the page switching configuration file of each display page may further include target page indication information. In this embodiment, the target page indication information can be used to indicate whether the display page is a target page. In this way, when a certain display page is displayed, if it is indicated as a target page, a screenshot can be taken.

[0103] Of course, if the page switching configuration file is shared by multiple display pages, the page switching configuration file can be designed to include the position information of the switching input box on each display page, the switching order of the target pages (the switching order of the target pages also belongs to a type of target page indication information), and the page switching paths of each target page (exemplarily, it can be a storage address).

[0104] For example, assume that there are 5 target pages among the above-mentioned multiple display pages, which are represented by A, B, C, D, and E respectively. The switching order of the target pages can be exemplarily: A -> C -> E -> B -> D, or, A -> (C, E) -> (B, D), (C, E) means that pages C and E are displayed at the same time. Of course, there can be other formats to represent the simultaneous display of multiple pages. Those skilled in the art can design it flexibly, such as C+E, which will not be elaborated here.

[0105] In this way, the next target page to be displayed can be determined based on the switching order of the target pages: assume that the current page is a non-target page and page A has been displayed, then it can be determined that page C or pages C and E are the next target pages; based on the position information of the switching input box on the current page, input the page switching path of the next target page (such as the storage addresses of pages C and E) into the switching input box, and according to the input page switching path, switch the page displayed on the industrial device from the current page to the next target page.

[0106] In another example, the page switching control may include a switching input box and a confirmation button. The page switching configuration file is set for each display page, which may include position information of the switching input box and the confirmation button on the corresponding display page, and the page switching path of the next target page. In this example, step 301 may include the following steps: Based on the position information of the switching input box, the page switching path of the next target page is input into the currently displayed page switching input box.

[0107] Based on the position information of the confirmation button, a click operation is performed on the confirmation button of the currently displayed page, and the display page of the industrial device can be switched from the current page to the next target page according to the input page switching path.

[0108] Then step 301 may exemplarily include the following steps: Step 3011: Based at least on the page switching control position information in the page switching configuration file, perform at least one click operation on the page switching button to switch from the current page to the next target page.

[0109] The following describes how to implement step 3011 in different situations.

[0110] In case 1, the industrial equipment is pre-configured with a page display sequence for displaying page by page.

[0111] Assume that the default page display order of the industrial control program of the industrial equipment is to display page by page from page 01 to page 11. In this case, the default page display order can be followed to switch to the next target page.

[0112] In one example, taking each display page as an example, the page switching configuration file may include the position information of the page switching button on the display page and the target page indication information (hereinafter referred to as indication information), and the indication information may be used to indicate whether the display page is the target page. Then step 3011 may include the following steps: Based on the position information of the switch button, click the switch button on the currently displayed page (assuming page 01) and switch to the next page (assuming page 02) according to the default page display order. If the next page (i.e. page 02) indicates that the page is the target page, you can take a screenshot and proceed to the next click operation and subsequent operations. If it is not the target page, directly proceed to the next click operation and subsequent operations: Based on the position information of the switch button on the currently displayed page (page 02), the switch button is clicked to switch to the next page (assuming it is page 03) according to the default page display order. If the next page (i.e. page 03) switched to indicates that the page is the target page, a screenshot operation can be performed, and the next click operation and subsequent operations can be performed. If it is not the target page, the next click operation and subsequent operations can be performed directly, and so on, which will not be repeated.

[0113] In another example, the following example can be used to implement switching to the next target page in the default page display order: Each display page corresponds to a page switching configuration file, which may include a page switching path and target page indication information. In this example, the page switching path refers to the path of the page switching button that needs to be clicked to switch from the current page to the next target page.

[0114] Furthermore, the above-mentioned page switching path can be a control switching coordinate sequence of the page required to jump from the current page to the next target page, wherein the coordinate sequence can include each page identifier and the corresponding page switching control position information, and the page identifier and the corresponding page switching control position information are bound one by one. In the process of switching to the next target page, the position information of each page switching control that needs to be clicked can form a click trajectory.

[0115] For example, assume that the default page display order of the industrial control program of the industrial equipment is to display page by page from page 01 to page 11. Page 03 and page 07 are the target pages.

[0116] For example, if the current page displayed is page 01, and its next target page is page 03, then the page switching configuration file corresponding to page 01 may include a page switching path from page 01 to page 03, which is determined according to the default page display order, and may be a control switching coordinate sequence, which may be in the form of A1(x1,y1)->A2(x2,y2), where A1 and A2 are page identifiers of pages 01 and 02, (x1,y1) are the coordinates of the page switching button on page 01, and (x2,y2) are the coordinates of the page switching button on page 02. By clicking on the coordinates (x1,y1) on page 01, it is possible to switch to page 02, and then by clicking on the coordinates (x2,y2) on page 02, it is possible to switch to page 03, thereby achieving switching to the target page 03. Of course, in some embodiments, the page identifier may not be reflected in the coordinate sequence.

[0117] For page 02, its next target page is also page 03, and its corresponding page switching configuration file may include coordinates A2 (x2, y2). Clicking the coordinates (x2, y2) can switch to page 03.

[0118] The next target page of target page 03 is page 07, and the page switching configuration file corresponding to target page 03 may include a page switching path for switching from page 03 to page 07. The page switching path is determined according to the default page display sequence, and may be a control switching coordinate sequence, which may be in the form of A3 (x3, y3) -> A4 (x4, y4) ... -> A6 (x6, y6). Clicking the coordinates on the control switching coordinate sequence one by one may switch to page 07.

[0119] As for target page 07, its next target page is 03, and its corresponding page switching configuration file may include a page switching path from page 07 to page 03. The page switching path is determined according to the default page display order, and may be a control switching coordinate sequence, which may be in the form of A7 (x7, y7) -> A8 (x8, y8) ... -> A11 (x11, y11), and by clicking (x11, y11) of page 11, you can switch back to page 01. And page 01 can switch all the way to target page 03 according to its page switching configuration file.

[0120] In other embodiments, the target page or the page switching configuration file of each page may further include target page indication information for indicating whether the corresponding page is the target page. If it is the target page, a screenshot may be taken.

[0121] In the case of multiple pages sharing a page configuration file, the page configuration file may include multiple page switching paths to achieve switching to different target pages. Assume that the default page display order of the industrial control program of the industrial equipment is to display page by page from page 01 to page 11. Among them, page 03 and page 07 are target pages, and the shared page configuration file may include three page switching paths, namely: A1(x1,y1)->A2(x2,y2); A3(x3,y3)->A4(x4,y4)…->A6(x6,y6); A7(x7,y7)->A8(x8,y8)…->A11(x11,y11).

[0122] In addition, target page indication information of each page may also be included, which is used to indicate whether the corresponding page is the target page.

[0123] Alternatively, the shared page configuration file exemplarily includes a page switching path: A1(x1,y1) -> A2(x2,y2) -> A3(x3,y3) … -> A11(x11,y11), and includes the target page indication information of each page, which is used to indicate whether the corresponding page is the target page. When switching to a certain page, it is possible to decide whether to take a screenshot according to the corresponding target page indication information.

[0124] In the second case, the industrial device is not pre-configured with a default page display order, or although it is configured with a default page display order, it does not follow this default page display order.

[0125] In the second case, there are various implementation methods for "switching to the next target page". For example: The first method: Taking the example that each page corresponds to a page configuration file, this file can exemplarily include the storage address of the target page to which it is required to jump and the position information of the switching button. After clicking, the next target page can be directly obtained according to this storage address and displayed, skipping the non-target pages.

[0126] The second method: Taking the example of a multi-page shared page configuration file, this file can exemplarily include: the page identifier of each display page, the position information of the switching button on this display page, and the storage address of the target page to which it is required to jump. According to the position information of the switching button on a certain display page, after clicking the switching button on it, the target page can be obtained and displayed according to the storage address of the target page to which it is required to jump.

[0127] It should be noted that the page identifier of each page, the position information of the switching button on this page, and the storage address of the target page to which it is required to jump can be reflected in a sequence. For example, still using the previous example (pages 03 and 07 are target pages), the sequence can exemplarily be: A1(x1, y1, M3) -> A3(x3, y3, M7) -> A7(x7, y7, M1 or M3). M3, M1, and M7 in the sequence are the storage addresses of pages 01, 03, and 07. Based on this sequence, by sequentially clicking on the coordinates (x1,y1), (x3,y3), (x7, y7), it is possible to directly jump from page 01 to page 03, then from page 03 directly to page 07, and then from page 07 directly to page 01 or page 03.

[0128] The third type: The page configuration file includes the page switching sequence and the control switching coordinate sequence (this sequence can be referred to the foregoing description). During the process of jumping from the current page to the next target page, exemplarily, first obtain the next page identifier according to the target page switching sequence, and obtain the page switching control position information corresponding to the next page identifier. Then, after clicking on the page switching control position information, load the next page, and repeat the above process until switching to the next target page for display and screenshot.

[0129] Further, during the page switching process, one, two or more pages can be displayed in the display area each time.

[0130] As an implementable way, taking the example that one page is displayed in the display area each time, the industrial control program inside the industrial device can control the industrial device to display the current page. The current page includes the current page switching button. Obtain the current page switching control position information corresponding to the current page switching button, click the current page switching button, determine the next target page according to the target page switching sequence, and then obtain the page switching path from the current page to the next target page. Switch the display page of the industrial device from the current page to the next target page, and perform a screenshot operation on the next target page to obtain the display image corresponding to the next target page. Among them, the current page can be a certain target page of the industrial device or a non-target page. A certain target page can be the initial target page or an intermediate target page in the target page switching sequence. If the current page is a target page, perform a screenshot operation on the current page to obtain the display image corresponding to the current page.

[0131] Exemplarily, taking the example that one page is displayed in the display area each time and the target page switching sequence is A→B→C→D, please refer to Figure 5As shown, after the industrial device is started up, the industrial control program inside the industrial device can control the industrial device to display the first target page A, that is, the initialization page. The target page A can include initialization information and a page switching button. Then, obtain the coordinates of the page switching control corresponding to the page switching button pre-configured on page A, for example, (a, b). Perform a click operation on the page switching button corresponding to (a, b). After the click operation is completed, determine the next target page as B according to the target page switching sequence. The B page can be, for example, the main operation interface. The main operation page includes operation information. Then, obtain the page switching path from page A to page B, and control the display page to switch from the current page A to the next target page B according to this page switching path. Similarly, the target page B can include a page switching button. Then, obtain the coordinates of the page switching control corresponding to the page switching button pre-configured on B, for example, (b, c). Then, perform a click operation on the page switching button corresponding to (b, c). After the click operation is completed, determine the next target page as C according to the target page switching sequence. The C page can be the production parameter page, the production process monitoring page. Then, in the same way as switching from A to B page, according to the page switching path corresponding to page B, switch from page B to page C and then from page C to page D. Among them, after switching to each target page, perform a screenshot operation to obtain the display images corresponding to each target page.

[0132] As another implementable way, taking the example that one or two pages are displayed in the display area each time, control the industrial device to display the current page through the industrial control program inside the industrial device. Assume that there is one page on the current page. Then, according to the coordinates corresponding to the switching button on the current page, click the switching button. Determine the two target pages to be displayed after switching (that is, including two next target pages) according to the target page switching sequence. Then, obtain the page switching path from the current page to the next target page, and switch the display page of the industrial device from the current target page according to the page switching path, display two target pages simultaneously, and perform a screenshot operation to obtain the display image corresponding to the next target page. Among them, if the current page includes two target pages of the industrial device, perform a screenshot operation on the two target pages to obtain the display images corresponding to the two target pages.

[0133] Exemplarily, taking the example that one or two pages are displayed in the display area each time and the target page switching sequence is A→B+C, please refer to Figure 6As shown, after the industrial device is started up, the industrial control program inside the industrial device can control the industrial device to display a target page A. The target page A or the display area of the industrial device may include a page switching button, and its coordinates are exemplarily (a, b). Then, a click operation is performed on the page switching button corresponding to (a, b). After the click operation is completed, the next two target pages are determined to be page B and page C according to the target page switching sequence. Then, the page switching paths for switching from page A to page B and page C are obtained, and the display area is controlled to switch from displaying the current page A to displaying the two target pages B and C according to the page switching paths. Among them, page B can be an operation information page 1, which contains operation information 1, and page C can be an operation information page 2, which contains operation information 2.

[0134] Among them, after determining that the target pages after switching are page B and page C, the page switching path of page B (the storage address of page B) and the page switching path of page C (the storage address of page C) can be obtained. Then, according to the storage address of page B and the storage address of page C, page B and page C are obtained, and page B and page C are displayed at different positions in the display area to display page B and page C simultaneously.

[0135] As another implementable manner, taking the display area displaying one or two pages each time as an example, the industrial control program inside the industrial device controls the industrial device to display the current page. The current page includes two display pages, and the combination modes of the two pages can include: two non-target pages, one non-target page and one target page, and two target pages. Then, based on the coordinates of the page switching control, the switching button on the current page is clicked, the next target page is determined according to the target page switching sequence, and then the page switching path for switching from the current page to the next target page is obtained to switch from the current target page to the next target page. The number of the next target pages can be one or two, and a screenshot operation is performed on the next target page to obtain the display image corresponding to the next target page.

[0136] Exemplarily, taking the display area displaying one or two pages each time and the target page switching sequence being A + B → C as an example, please refer to Figure 7As shown, when the industrial equipment is started, the industrial control program inside the industrial equipment can control the industrial equipment to display two target pages A and target page B. A page switching button can be set on the display screen (display area) of the industrial equipment, and its coordinates are exemplarily (a, b). Then, a click operation is performed on the page switching button corresponding to (a, b). After the click operation is completed, the next target page is determined to be page C according to the target page switching sequence, and then the page switching path from page A or page B to page C is obtained. According to the page switching path, the page displayed on the display area is controlled to switch from the current page A and page B to the next target page C. Among them, page A can be an initialization page, which contains initialization information, page B can be a status information page, which contains status information, and page C can be an operation information page 1, which contains operation information 1. The page switching path can be exemplarily the storage address of page C.

[0137] As another possible implementation method, taking the display area displaying multiple pages at a time as an example, the industrial control program inside the industrial equipment can be used to control the industrial equipment to display the current page, and the current page includes two pages, and the two pages may include: two non-target pages, one non-target page and one target page, and two target pages. A page switching button can be set on the display screen (display area) of the industrial equipment, and then the coordinates of the page switching button are obtained, and the current page switching button is clicked. The target page after switching is determined according to the target page switching sequence, and then the page switching path from the current page to the target page after switching is obtained, and the page displayed in the display area of ​​the industrial equipment is switched from the current two pages to the two target pages, and the target page after switching is screenshotted, and the display image corresponding to the target page after switching is obtained. Among them, if the current page includes two target pages of the industrial equipment, the two target pages are screenshotted to obtain the display images corresponding to the two target pages.

[0138] For example, the display area displays two pages at a time, and the target page switching order is A+B→C+D. Figure 8As shown, after the industrial device is started up, the industrial control program inside the industrial device can control the industrial device to display two target pages A and B simultaneously. The target page A or its display area may include a page switching button, whose coordinates are exemplarily (a, b). Then, a click operation is performed on the page switching button corresponding to (a, b). After the click operation is completed, the next two target pages are determined to be page C and page D according to the target page switching sequence. Then, the page switching path from page A and page B to page C and page D is obtained, and the display page is controlled to switch from the current pages A and B to the next two target pages C and D according to this page switching path. Among them, page A can be an initialization page containing initialization information, page B can be a status information page containing status information, page C can be an operation information page 1 containing operation information 1, and page D can be an operation information page 2 containing operation information 2.

[0139] In this embodiment, according to the page switching configuration file, the display page of the industry can be accurately controlled to switch from the current page to the next, two or more target pages, so as to automatically realize the flexible switching of the display page, improve the flexibility of page switching, and after switching to the next, two or more target pages, a display image is obtained through a screenshot operation, and then a more comprehensive and accurate display image of the industrial device is obtained, improving the accuracy of image acquisition.

[0140] The following introduces how to perform image segmentation.

[0141] In one of the embodiments, the above-mentioned preprocessed image is subjected to image segmentation processing to obtain sub-images, and a target area containing character information is determined from the sub-images, including: According to the image segmentation configuration file, the preprocessed image is subjected to image segmentation processing to obtain sub-images containing character information; the background area in the sub-images is removed to obtain a target area containing character information.

[0142] It can be understood that characters generally do not fill the entire page. Therefore, the preprocessed image needs to be subjected to image segmentation processing to obtain sub-images containing character information, and the background area in the sub-images is removed to obtain a target area containing character information.

[0143] The above-mentioned image segmentation configuration file is a file required for pre-configured image segmentation processing, and can be custom-set according to actual needs.

[0144] Exemplarily, each target page can correspond to an image segmentation configuration file.

[0145] Examples of segmentation methods based on image segmentation profiles can be applied to the case of segmenting the display images of single-mode display industrial devices. This is because single-mode display industrial devices have a single display screen, and their display content and mode are relatively fixed. Therefore, segmentation can be performed through a pre-set image segmentation profile. Of course, for multi-mode display industrial devices, if the display content and mode of each display page are fixed, an image segmentation profile can also be configured for each display page or the display pages of interest.

[0146] The regional image segmentation profile includes at least one of the following: regional segmentation position information, regional parameter information, and regional description information.

[0147] Among them, the regional segmentation position information can be used to characterize the position of the area to be segmented, or the position for segmentation, etc. The regional segmentation position information may include regional segmentation coordinates, which may include abscissa and ordinate. The regional segmentation coordinates can include one or more.

[0148] For example, if the regional segmentation coordinates include two, namely (A, B) and (C, D), then the segmentation line can be determined according to (A, B) and (C, D) (the line connecting the two coordinate points forms the segmentation line), and the pre-processed image can be initially segmented according to the segmentation line to obtain two image areas; For another example, the regional segmentation position information may include the coordinates of the center of the circle, and in addition, the length of the diameter or radius may also be included. The pre-processed image can be segmented according to the center coordinates, diameter or radius length to obtain a circular area.

[0149] For another example, the regional segmentation position information may include the coordinates of the four vertices of a rectangle; or, it includes the coordinates of one vertex of the rectangle and the length and width of the rectangle. The pre-processed image can be segmented according to this regional segmentation position information to obtain a rectangular area.

[0150] For another example, the regional segmentation position information includes the coordinates of the two foci of an ellipse and the lengths of the major and minor axes. The pre-processed image can be segmented according to this regional segmentation position information to obtain an elliptical area.

[0151] The above-mentioned regional description information includes information describing the target area. The regional description information is usually some text descriptions and can be used to determine the nature of the target area, such as "the target is a circular red object"; the above-mentioned regional parameter information includes parameter information in the target area, and exemplary ones can include color range, texture features, etc., which are used to clarify the characteristics of the segmented area.

[0152] The following introduces how to use the regional segmentation position information, regional parameter information, and regional description information to jointly segment the target area: After obtaining the preprocessed image, preliminary image segmentation can be performed on the preprocessed image based on the region segmentation position information to obtain corresponding image regions. Then, refined segmentation is performed on the preliminarily segmented image regions according to the region parameter information: for example, if the region parameter information includes a color range, color thresholding can be used for segmentation; for another example, when the region parameter information includes texture feature information, methods such as Gabor filters can be used for texture analysis and segmentation to obtain the refined segmented image regions. Finally, the segmentation result after refined segmentation is further adjusted according to the region description information to obtain a sub-image containing character information. Among them, during the process of adjusting the segmentation result after refined segmentation according to the region description information, if the region description information indicates that the target is a circular object, the Hough transform can be used to detect the circular regions in the segmentation result after refined segmentation, and the regions containing character information are screened out and used as sub-images. After obtaining the sub-image, the background region in the sub-image is determined and removed to obtain the target region containing character information.

[0153] In another embodiment, the above-mentioned image segmentation process for the preprocessed image to obtain a sub-image and determining the target region containing character information from the sub-image includes: inputting the preprocessed image into a trained image segmentation model for image segmentation processing to obtain the target region containing character information output by the image segmentation model; the image segmentation model is trained according to historical display images and corresponding region annotation results.

[0154] Exemplarily, the above-mentioned image segmentation model is a pre-trained model for image segmentation, which can exemplarily include multiple neural network layers, such as convolutional layers, pooling layers, deconvolutional layers, and output layers, and the neural network layers can be connected in sequence. In an exemplary embodiment, the above-mentioned image segmentation model can be a U-Net model, FCN (Fully Convolutional Network), Mask R-CNN, etc.

[0155] Specifically, after obtaining the preprocessed image, the preprocessed image can be input into a trained image segmentation model, and feature extraction is performed through multiple convolutional layers in sequence to capture different features in the image, such as edges, textures, etc. Each convolutional layer will generate multiple feature maps, and these feature maps contain feature information of the image at different scales and directions. Then, the feature maps are processed through a pooling layer, usually using max pooling or average pooling operations, to obtain low-resolution feature maps. The purpose is to reduce the spatial resolution of the image, reduce the amount of data, and at the same time retain important feature information. The pooling operation can make the model more robust to small changes in the image and can expand the receptive field of the model, that is, the model can "see" a larger range of the image area. Then, deconvolution operations are performed through a deconvolution layer to upsample the low-resolution feature maps to the resolution of the original preprocessed image. The deconvolution operation is actually the inverse process of convolution. It learns how to restore the low-resolution feature maps to high resolution to obtain high-resolution feature maps, thereby achieving the restoration of the image resolution. During the deconvolution process, the spatial size of the feature maps will gradually increase, while the number of channels of the feature maps will decrease. Then, the high-resolution feature maps are concatenated with the corresponding-level feature maps mentioned above through a concatenation layer to fuse feature information at different scales and obtain fused feature maps. The fused feature maps contain the underlying detail information and semantic information of the image. Through concatenation, these information can be fully utilized to improve the accuracy of segmentation. Finally, the fused feature maps are passed through an output layer to obtain the final segmentation result, that is, the target region of character information.

[0156] In one embodiment, in order to accurately determine the target region containing character information, the initial model can be trained to construct an image segmentation model. Figure 9 An exemplary process for constructing an image segmentation model is shown, including the following steps S401 to S405: Step S401, obtain historical display images and divide the historical display images into a training set and a test set; the historical display images are marked with regional annotation results.

[0157] The above training set refers to the samples used for training the initial model, and the test set refers to the samples used for testing the image segmentation model to be tested, which is used to evaluate the performance of the image segmentation model to be tested. The data of the test set has the same distribution characteristics as the data of the training set to ensure the effectiveness of the evaluation.

[0158] It should be noted that the above historical display images can be display images covering various different scenarios, lighting conditions, object postures, etc. The above regional annotation results are the regions containing character information annotated in the historical display images, that is, the target regions in the historical display images. Boxes of various shapes can be used to circle the character information to obtain the regional annotation results. The regional annotation results can be completed manually by professionals or can be annotated with the help of some semi-automatic annotation tools to improve efficiency.

[0159] In the embodiment of the present application, when the training set and the test set have been established, step 401 can be skipped.

[0160] Step S402, use the initial model to perform segmentation processing on the historical display images in the training set to obtain regional prediction results.

[0161] Step S403, according to the regional prediction results and the corresponding regional annotation results, calculate the loss function, and adjust the model parameters of the initial model using an iterative algorithm according to the minimization of the loss function until the loss function reaches the minimum or reaches the number of iterations, to obtain the image segmentation model to be tested.

[0162] Step S404, input the historical display images in the test set into the image segmentation model to be tested to obtain the corresponding regional segmentation results.

[0163] Step S405, determine the corresponding evaluation data based on the regional segmentation results and the regional annotation results corresponding to the test set. When the evaluation data meets the preset conditions, determine the image segmentation model to be tested as the image segmentation model.

[0164] The above evaluation data includes accuracy rate, recall rate, F1 value, etc.

[0165] Based on steps 401-405, a more detailed construction method includes: after obtaining the historical display images, divide the historical display images into a training set and a test set according to a preset ratio. For example, 80% of the historical display images are used as the training set, and 20% of the historical display images are used as the test set. Then set hyperparameters, such as setting the learning rate, the number of iterations, etc. After the hyperparameters are set, input the training set into the initial model with the parameters set. After passing through multiple convolutional layers, pooling layers, deconvolutional layers, splicing layers, and output layers, obtain the regional prediction results. According to the regional prediction results and the corresponding regional annotation results, calculate the loss function, and adjust the model parameters (weights, biases) of the initial model using an iterative algorithm according to the minimization of the loss function until the loss function reaches the minimum or reaches the number of iterations, to obtain the image segmentation model to be tested. Among them, the loss function is an index used to measure the difference between the regional prediction results and the regional annotation results.

[0166] After obtaining the image segmentation model to be tested, the historical display images in the test set are input into the image segmentation model to be tested to obtain the corresponding region segmentation results. The corresponding evaluation data is determined based on the region segmentation results and the region annotation results corresponding to the test set. When the evaluation data does not meet the preset conditions, one or more methods such as adjusting the hyperparameters of the initial model, adjusting the model structure, and performing data enhancement on the sample data of the training set can be used, and then retraining is performed until the evaluation data meets the preset conditions. When the evaluation data meets the preset conditions, for example, when the accuracy is greater than the preset accuracy threshold, the image segmentation model to be tested that meets the preset conditions is used as the image segmentation model.

[0167] Wherein, adjusting the model parameters of the initial model may be to update matrix parameters such as a weight matrix and a bias matrix in the initial model. Wherein, the weight matrix and the bias matrix include but are not limited to matrix parameters in each layer in the initial model. In this embodiment, the model parameters in the initial model are updated to achieve the purpose of training the initial model.

[0168] It can be understood that when the model parameters in the initial model are updated by the loss function, it can mean that when the initial model has not converged, the model parameters of the initial model are adjusted so that the initial model is in a converged state, thereby obtaining the image segmentation model to be tested. The initial model is in a converged state, which can mean that the difference between the region prediction result and the region annotation result is less than a preset threshold, or the rate of change of the difference between the region prediction result and the region annotation result approaches a certain lower value.

[0169] In the embodiment of the present application, by training the image segmentation model, it is possible to directly perform image segmentation on the preprocessed image through the image segmentation model, accurately determine the target area containing character information, thereby improving the accuracy of target area determination.

[0170] The following uses the OCR recognition algorithm as an example to introduce the generation of industrial data.

[0171] In one embodiment, the aforementioned step 204 may illustratively include the following steps: The target area is identified and processed using an OCR algorithm to obtain text data; it is determined whether there is a historical text corresponding to the text data; when there is a historical text, the text data is compared with the historical text according to data correction rules, and the industrial data corresponding to the target area is determined based on the comparison results.

[0172] It should be noted that since there may be differences in angles and lighting conditions during the process of obtaining the display image, in order to ensure the accuracy of text recognition, the display image of the same target interface in the display area can be obtained multiple times (which can refer to taking multiple pictures during one acquisition process), and then the display image is subjected to recognition processing to obtain the corresponding industrial data.

[0173] The above text data may include: characters, numbers, letters, etc.

[0174] The historical text can be data that has been corrected after being recognized and processed, and whose acquisition time is before the acquisition time of the text data.

[0175] In one example, during the process of determining whether there is a historical text corresponding to the text data, it can be determined exemplarily according to the page identifier: query whether there is a historical text in the database that matches the page identifier of the current text data. If it exists, it indicates that there is a historical text corresponding to the text data, and the text corresponding to the page identifier is used as the historical text corresponding to the text data; if it does not exist, it indicates that there is no historical text corresponding to the text data. The page identifier of the text data here refers to the page identifier corresponding to the display image to which the text data belongs. Similarly, the page identifier of the historical text refers to the page identifier corresponding to the display image to which the historical text belongs.

[0176] In another example, during the process of determining whether there is a historical text corresponding to the text data, it can be judged based on text similarity. By counting the number of occurrences of each word in the text data and the historical text, using cosine similarity and Jaccard similarity algorithms, the similarity between the text data and the historical text is calculated according to the number of occurrences, and the text with a similarity greater than the preset similarity threshold is used as the historical text.

[0177] Suppose the text data of the industrial equipment is "The current temperature of Equipment A is 80°C, the pressure value is 1.2 MPa, and the operating status is normal", and the historical text is: "The current temperature of Equipment A is 78°C, the pressure value is 1.1 MPa, and the operating status is normal". The preset similarity threshold is 0.6. The word frequency of the text data is counted to obtain: {"Equipment A": 1, "current": 1, "temperature": 1, "80°C": 1, "pressure value": 1, "1.2 MPa": 1, "operating status": 1, "normal": 1}, and the word frequency of the historical text is counted to obtain {"Equipment A": 1, "current": 1, "temperature": 1, "78°C": 1, "pressure value": 1, "1.1 MPa": 1, "operating status": 1, "normal": 1}. Using the similarity algorithm, the similarity between the text data and the historical text is obtained. Suppose the calculated similarity is 0.75. Since 0.75 > 0.6 (the preset similarity threshold), the historical text can be used as the historical text corresponding to the current text data.

[0178] In other embodiments of the present application, when there is no historical text corresponding to the text data, a preset standard text and corresponding data correction rules (which can be called standard data correction rules) are obtained. The standard data correction rules can exemplarily include at least one of a standard data format, a standard data fluctuation range, and a standard parameter type. The text data can be compared with the preset standard text according to the standard data correction rules. For example, it can be checked whether the text data conforms to the standard data format, the standard parameter type, and the standard data fluctuation range to obtain a comparison result. For text data that does not conform to the standard data correction rules, it can be used as data to be corrected for correction processing to obtain industrial data.

[0179] The following introduces comparison and correction.

[0180] The comparison method can be one or more of comparing based on the text content or comparing based on the text structure, so as to obtain a comparison result. The comparison result of each comparison method is used to characterize whether the text data is consistent with the historical text. Subsequently, the industrial data corresponding to the target area can be determined according to the comparison result. It should be noted that when multiple comparison methods are used for comparison, it should be finally determined that the comparison is consistent only when the comparison results of all comparison methods indicate consistency. If they are inconsistent, the text data can be corrected or discarded as appropriate, or manual correction can also be introduced.

[0181] When comparing based on the text content, the character set in the text data and the character set in the historical text can be counted, and the two character sets can be compared. When the character sets are compared and consistent, it indicates that the text data is consistent with the historical text; otherwise, it indicates that the text data is inconsistent with the historical text. The text content with inconsistent comparison needs to be used as the text to be corrected for correction processing. It can also be a comparison such as the data fluctuation range.

[0182] When comparing based on the text structure, it can be determined whether the data format in the text data conforms to the corresponding data format of the historical text. For data that does not conform to the data format, it needs to be used as the text to be corrected for correction processing.

[0183] For example, assume that for the text data "The current temperature of device A is 85°C, the pressure value is 1.3 MPa, and the operating state is normal", its character set is counted, which includes various characters such as Chinese characters (device, current, temperature, pressure value, operating state, normal), English letters (A, M, P, a), numbers (85, 1, 3), punctuation marks (,, °C, ). For the historical text A "The current temperature of device A is 85°C, the pressure value is 1.3 MPa, and the operating state is normal", its character set is also counted, which also includes the corresponding Chinese characters, English letters, numbers, punctuation marks, etc. The character set of the text data is compared with the character set of the historical text A "The current temperature of device A is 85°C, the pressure value is 1.3 MPa, and the operating state is normal". It can be found that the character sets of the two are exactly the same, which indicates that the text data is consistent with this historical text A in comparison.

[0184] If the historical text A is "The current humidity of device A is 85 g / m 3 , the pressure value is 1.2 MPa, and the operating state is normal", and the text data is "The current temperature of device A is 84 g m 3 , the pressure value is 1.3 MPa, and the operating state is normal", then when the character sets of the above text data and the historical text A are compared, due to the "temperature" and "g m 3 " in the text data being different from the corresponding characters of "humidity" and "g / m 3 " in the historical text A, the two character sets are inconsistent, which indicates that the text data is inconsistent with this historical text A in comparison. The inconsistent part is used as the text to be corrected for correction processing.

[0185] Taking the comparison with the historical text and correction as an example, in one example, the aforementioned data correction rules may include: data format. This data format refers to the specific structure and expression method adopted during data storage, transmission, and presentation. It stipulates the organization form of data, coding method, and the arrangement order of each part of the data, etc.

[0186] In this example, the above "According to the data correction rules, compare the text data with the historical text, and determine the industrial data corresponding to the target area according to the comparison result" may include: Match the data format of the text data with the historical text.

[0187] If the match fails, regard the text data as the text to be corrected; according to the data correction rules, perform correction processing on the text to be corrected to obtain the industrial data corresponding to the target area.

[0188] Exemplarily, when the industrial device is a switch, obtaining the display image of the switch and identifying and determining the text data in the target area of the display image may include "on" or "off". It can be judged whether there is historical text corresponding to the text data. When there is historical text, the text data is compared with the historical text. By obtaining the data format of the historical text, for example, in the historical text, the data format corresponding to the switch is recorded as 1 and 0, then the text data is inconsistent with the historical text, indicating that the data format of the text data does not conform to the data format in the historical text, and the "on" or "off" that does not conform to the data format is used as the text to be corrected. Then, according to the data correction rule, "on" is recorded as 1 and "off" is recorded as 0.

[0189] For another example, when the industrial device is an instrument, in the process of judging whether the text data conforms to the data format corresponding to the historical text. For example, when the text data is "202A-1O-L6", and it is determined that the data format corresponding to the historical text is the date format, the date format is usually fixed (such as YYYY year, MM / DD / YYYY, DD-MM-YYYY, etc.). It can be checked whether the digital sequence in the text contains non-numeric characters. If non-numeric characters are found, they are replaced with the most likely numeric characters according to the context. Therefore, by checking whether non-numeric characters are included, if so, it is determined that the data format of the text data does not conform to the data format corresponding to the historical text, and the text data "202A-1O-L6" that does not conform to the date format is used as the text to be corrected. According to the data correction rule, the content with letters in the text to be corrected is replaced with the corresponding numbers. For example, as shown in Figure 10 shown, replace "A" with "4", "L" with "1", and "O" with "0" in it, so as to obtain the corresponding industrial data "2024-10-16". In addition, in other embodiments of the present application, pattern matching can also be used for correction. Still taking the date format as an example, some common patterns can be predefined, for example: the year is usually a four-digit number (such as 2024), and the date is usually in a specific format (such as MM / DD / YYYY, DD-MM-YYYY). When using pattern matching, it can be checked whether the digital sequence in the text data conforms to the predefined pattern. If it does not conform to the pattern, it is replaced with the most likely numeric character according to the context analysis. For example, for "2O2A", which does not conform to the pattern of four-digit numbers, it can be replaced with "2024" according to the context analysis.

[0190] For a date or a digital sequence, the context analysis can be based on the following rules: the digital sequence usually only contains numeric characters (0-9) and does not contain letters; the date format is usually fixed (such as YYYY year, MM / DD / YYYY, DD-MM-YYYY, etc.).

[0191] Another example is temperature data, which may be in the format of XX°C, XX°F. Temperature data may have format problems due to OCR recognition errors, such as missing spaces between numbers and units (such as 25°C is recognized as 25C), and units may be incorrectly recognized (such as °C is recognized as OC or oC). Correction can be performed based on its format. Exemplary correction methods include the following: Adding spaces: Check whether there is a missing space between numbers and units, and add it if necessary; Unit correction: Use a predefined unit list (such as °C, °F) to correct incorrect units.

[0192] In addition, unit conversion can be performed. If the units of temperature data in the text data are inconsistent (such as mixing Celsius and Fahrenheit), or the units in the text data are inconsistent with the units in the historical text, unit conversion can be performed to unify the format. The conversion formula is as follows: Celsius to Fahrenheit: F=C×59+32; Fahrenheit to Celsius: C = (F − 32) × 95; For example, the temperature in the text data is: 77°F, which is converted to 25°C.

[0193] In addition to unit correction, temperature format correction also includes number correction. OCR may misidentify numbers, such as identifying 25 as 2S or Z5. For temperature data, correction can be done by identifying and replacing common confusing pairs of numbers and letters (such as 0 and O, 1 and I, 5 and S).

[0194] In one example, the data correction rule further includes: at least one parameter type and an upper and lower fluctuation range of the data corresponding to each parameter type.

[0195] Before or after determining whether the data format of the text data conforms to the data format corresponding to the historical text, the method further includes: Obtain the upper and lower fluctuation range of the text data relative to the historical text data as the target fluctuation range; According to the parameter type to which the text data belongs, the target fluctuation range of the text data is matched with the upper and lower fluctuation ranges of the data corresponding to the parameter type.

[0196] It should be noted that the above target fluctuation range can be calculated based on the text data and the historical text, and is used to characterize the difference between the text data and the historical text. Each parameter type can include one or more data upper and lower fluctuation ranges.

[0197] Specifically, when the target fluctuation range matches the data up and down fluctuation range of the corresponding parameter type, determine whether the text data conforms to the data format corresponding to the parameter type; use the text data that does not conform to the data format as the text data to be corrected; according to the data correction rule, perform correction processing on the text data to be corrected to obtain the industrial data corresponding to the target area.

[0198] Of course, if both the parameter type and the fluctuation range match, no correction is performed, and the text data is directly used as industrial data.

[0199] Exemplarily, for example, when the text data is temperature data with a corresponding value of 100 and the corresponding value of the historical text is 90, the data up and down fluctuation range of the text data relative to the historical data is determined to be an upward increase of 10, and this is used as the target fluctuation range. Then, obtain the parameter type to which the text data belongs and the corresponding data up and down fluctuation range, and determine whether the target fluctuation range of the text data matches the data up and down fluctuation range corresponding to the parameter type. For example, the up and down fluctuation range is 0~15 or -15~0, where 0~15 represents an upward increase of 0~15, and -15~0 represents a downward decrease of 0~15. It can be seen that the target fluctuation range of the text data matches the data up and down fluctuation range corresponding to the parameter type. Then, determine whether the text data conforms to the data format corresponding to the historical data. When it does not conform, use the text data that does not conform to the data format as the text data to be corrected, and according to the data correction rule, perform correction processing on the text data to be corrected to obtain the industrial data corresponding to the target area. When the text data conforms to the data format corresponding to the historical data, use the text data as the industrial data corresponding to the target area. When the target fluctuation range of the text data does not match the data up and down fluctuation range corresponding to the parameter type, the data can be filtered; the text data can also be used as abnormal data and an alarm prompt signal can be generated and sent to the central control system or the data center to give an alarm prompt for the abnormal data; the number of occurrences of the abnormal data can also be counted for subsequent data analysis. The alarm prompt signal can be a signal in the form of a sound alarm prompt, a color alarm prompt, or an image alarm prompt. This embodiment does not limit the manner of the alarm prompt signal in any way.

[0200] Further, after obtaining the industrial data corresponding to the industrial equipment, the industrial data can be stored in a database, or can be transmitted through a network to the central control system or the data center for further processing and analysis.

[0201] Optionally, for some special complex industrial equipment, the display area corresponding to a single-mode display with a single display type may also include the display area corresponding to a multi-mode display with multiple display types. In the process of obtaining the display images of each display area, screenshots of the display area corresponding to the multi-mode display can be taken through an industrial control program to obtain the display image corresponding to the multi-mode display, and the display area corresponding to the single-mode display can be image-captured through an industrial camera to obtain the display image corresponding to the single-mode display. Then, the OCR processing module inside the industrial equipment preprocesses, segments, recognizes, and corrects both the display image corresponding to the multi-mode display and the display image corresponding to the single-mode display to obtain the industrial data corresponding to each display area, and the industrial data corresponding to each display area can be sorted out to obtain the industrial data corresponding to the entire industrial equipment.

[0202] In this embodiment, by using the OCR algorithm to identify and process the target area to obtain text data and determining whether there is a historical text corresponding to the text data, when there is a historical text, it can be compared with the historical text, so as to comprehensively consider the content of the historical text, more comprehensively correct the error text that may appear in the text data, and thus ensure the accuracy and reliability of the industrial data. And it can be compatible with a variety of industrial equipment and industrial control systems. Whether data is collected through an industrial camera or through screenshot operations of the industrial control program, the corresponding display images can be collected, which has wide applicability. At the same time, through automated data collection and processing, the dependence on high-precision equipment and professional personnel is reduced, the equipment procurement and maintenance costs are reduced, and the errors caused by manual operations are reduced.

[0203] As mentioned in the background technology, the traditional technology collects the data of the display area through the communication protocol inherent in the industrial equipment. This data includes: equipment status, production capacity, alarms, industrial parameters, operation information, etc. However, this method requires complete dependence on the equipment manufacturer to provide communication interfaces and complete data definitions. If the equipment manufacturer does not support or the support is insufficient, data collection will face difficulties. After an enterprise purchases equipment, it often needs to spend additional funds to purchase communication interface licenses, thus increasing the enterprise's cost investment. And for some old equipment, due to the weak processing ability of the control system, a collection gateway needs to be installed to achieve data collection, cleaning, and processing in the equipment. After installing the collection gateway, the operation efficiency of the equipment will be affected, and even the normal production of the equipment will be affected, resulting in the inability to achieve data collection even after installing the collection gateway.

[0204] The industrial data recognition method provided by this application does not need to rely on the communication interfaces and data definitions provided by equipment manufacturers. Whether the equipment has a communication interface or not, and regardless of whether the equipment manufacturer provides complete support, as long as the display area on the equipment can be captured by an industrial camera or the data display interface can be intercepted, data collection can be achieved, getting rid of the excessive dependence on equipment manufacturers and eliminating the need to spend extra money to purchase communication authorizations, thus reducing the data collection cost of enterprises. Moreover, for old equipment, there is no need to install a gateway, and data can be collected by means of an industrial camera photographing the equipment or intercepting the data interface integrated in the equipment system, solving the problem of data collection for old equipment.

[0205] Based on the same inventive concept, the embodiment of this application also provides an industrial data recognition device for implementing the above-mentioned related content. The solution provided by this device to solve the problem is similar to the solution described in the above method. Therefore, the specific limitations in one or more embodiments of the industrial data recognition device provided below can refer to the limitations on the industrial data recognition method in the above text, and will not be repeated here.

[0206] In an exemplary embodiment, as Figure 11 shown, an industrial data recognition device is provided, including: An acquisition module 510, configured to acquire a display image of an industrial device; A preprocessing module 520, configured to preprocess the display image to obtain a preprocessed image; A segmentation module 530, configured to perform image segmentation processing on the preprocessed image to obtain sub-images, and determine a target area containing character information from the sub-images; A correction module 540, configured to recognize the target area to obtain text data, and perform correction processing on the text data to generate industrial data corresponding to the industrial device.

[0207] In an exemplary embodiment, the acquisition module 510 is configured to: When the device type of the industrial device is a multi-mode display industrial device, perform a screenshot operation on the target page of the industrial device through the industrial control program inside the industrial device to obtain a display image.

[0208] In an exemplary embodiment, in terms of performing a screenshot operation on the target page of the industrial device, the acquisition module 510 is configured to: Display each target page in the display area of the industrial device and perform a screenshot operation.

[0209] In an exemplary embodiment, in terms of displaying each target page in the display area of the industrial device, the acquisition module 510 is configured to: Perform at least one interaction operation on the page switching control at least according to the page switching control position information in the page switching configuration file, so as to switch from the current page to the next target page.

[0210] For the detailed content, reference can be made to the foregoing description and will not be elaborated here.

[0211] In an exemplary embodiment, in terms of performing image segmentation processing on the preprocessed image to obtain sub-images and determining a target region containing character information from the sub-images, the segmentation module 530 is used for: Perform image segmentation processing on the preprocessed image according to the image segmentation configuration file to obtain sub-images containing character information; remove the background region in the sub-images to obtain a target region containing character information; the image segmentation configuration file includes at least one of the following: region segmentation position information, region parameter information, and region description information; the region parameter information includes parameter information in the target region, and the region description information includes information describing the target region; or, Input the preprocessed image into a trained image segmentation model for image segmentation processing, and obtain a target region containing character information output by the image segmentation model; the image segmentation model is trained according to historical display images and corresponding region annotation results.

[0212] In an exemplary embodiment, the image segmentation model is constructed through the following steps: Obtain historical display images and divide the historical display images into a training set and a test set; the historical display images are marked with region annotation results; Use the initial model to perform segmentation processing on the historical display images in the training set to obtain region prediction results; According to the region prediction results and the corresponding region annotation results, calculate the loss function, and adjust the model parameters of the initial model using an iterative algorithm according to the minimization of the loss function until the loss function reaches the minimum or the number of iterations is reached, to obtain a to-be-tested image segmentation model; Input the historical display images in the test set into the to-be-tested image segmentation model to obtain corresponding region segmentation results; Determine corresponding evaluation data based on the region segmentation results and the region annotation results corresponding to the test set. When the evaluation data meets the preset conditions, determine the to-be-tested image segmentation model as the image segmentation model.

[0213] In an exemplary embodiment, in terms of identifying the target region to obtain text data and performing correction processing on the text data to generate industrial data corresponding to the industrial device, the correction module 540 is used for: Adopt the optical character recognition OCR algorithm to perform recognition processing on the target region to obtain text data; Determine whether there is historical text corresponding to the text data; the historical text is data obtained by recognition processing with the collection time before the collection time of the text data. When there is historical text, according to the data correction rule, compare the text data with the historical text, and determine the industrial data corresponding to the target area according to the comparison result.

[0214] In an exemplary embodiment, in terms of comparing the text data with the historical text according to the data correction rule and determining the industrial data corresponding to the target area according to the comparison result, the correction module 540 is used for: Match the data formats of the text data and the historical text; If the matching fails, regard the text data as the text to be corrected; According to the data correction rule, perform correction processing on the text to be corrected to obtain the industrial data corresponding to the target area.

[0215] In an exemplary embodiment, in terms of comparing the text data with the historical text according to the data correction rule and determining the industrial data corresponding to the target area according to the comparison result, the correction module 540 is further used for: Obtain the data fluctuation range of the text data relative to the historical text as the target fluctuation range; According to the parameter type to which the text data belongs, match the target fluctuation range of the text data with the data fluctuation range corresponding to this parameter type.

[0216] In another exemplary embodiment, in terms of performing a screenshot operation on the target page of the industrial device, the acquisition module 510 is used for: When the device type of the industrial device is a multi-mode display industrial device, perform a screenshot operation on all display pages of the industrial device through the industrial control program inside the industrial device, and filter out non-target pages in all screenshots to obtain a display image.

[0217] In an exemplary embodiment, in terms of obtaining the display image of the industrial device, the acquisition module 510 is used for: When the device type of the industrial device is a single-mode display industrial device, obtain the image obtained by the industrial camera performing an image acquisition operation on the display area of the industrial device.

[0218] The industrial camera can perform automatic acquisition or acquisition according to the instruction of the acquisition module 510.

[0219] The industrial data identification device can be applied to the server 104, the host computer or the industrial equipment in the form of software or hardware. When it exists in the form of software, the industrial data identification device can be a separate application program or a component of an application program or an operating system. When it exists in the form of hardware, the industrial data identification device can be a processor of the server 104, the host computer or the industrial equipment.

[0220] It can be seen that the industrial data identification device provided in the embodiment of the present application does not need to install a data acquisition gateway on the industrial equipment, thus avoiding the problem of affecting the equipment operation efficiency. It also does not need to implement data acquisition through the communication protocol inside the industrial equipment. It can automatically obtain the display image of the industrial equipment and preprocess the display image to obtain the preprocessed image, thereby providing good data guidance information for subsequent data recognition, and performing image segmentation processing on the preprocessed image to accurately obtain sub-images, so that the target area containing character information can be determined from each sub-image in a finer granularity, and then the target area is identified to obtain text data, and then the text data is corrected, which effectively reduces recognition errors and further improves the accuracy of industrial data recognition.

[0221] In an exemplary embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0222] The computer device may be a server or a terminal, and may be used as the server, host computer, edge design device or industrial device. Its internal structure diagram may be shown as follows: Figure 12 As shown, it includes: a processor, a memory, an input / output interface (Input / Output, referred to as I / O), a communication interface and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface.

[0223] Among other things, a bus may include a pathway for transmitting information between various components of a computer system.

[0224] The processor of the computer device is used to provide computing and control capabilities. The processor can be a general-purpose processor, such as a general-purpose central processing unit (CPU), a network processor (NP for short), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the solution of the present invention. It can also be a digital signal processor (DSP), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The processor may include a main processor (CPU). The number of processors can be one or more.

[0225] The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, computer programs, databases, programs for implementing the technical solution of the present invention, etc., and can also store the operating system and other critical services. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database of the computer device is used to store data in the page switching configuration file, image segmentation configuration file, etc. Among them, the above program may include program code, and the program code includes computer operation instructions. More specifically, the memory may include non-volatile memory, such as read-only memory (ROM), disk memory, flash, and other types of static storage devices that can store static information and instructions. In addition, it may also include random access memory (RAM), and other types of dynamic storage devices that can store information and instructions.

[0226] The input / output interface of the computer device is used to exchange information between the processor and external devices (generally peripheral devices).

[0227] The communication interface of the computer device is used to communicate with an external terminal through a network connection. The communication interface may include any device of the transceiver type to communicate with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area network (WLAN), etc.

[0228] When the computer program is executed by the processor, it realizes an industrial data recognition method. The functions of each unit in the industrial data recognition device provided in the above embodiments can be realized by the processor of the computer device executing the program stored in the memory and calling other devices.

[0229] In some embodiments, the above computer device may further include an input device and an output device. Among them, the input device may exemplarily include devices for receiving data and information input by the user, such as a keyboard, a mouse, a camera, an industrial camera module, a scanner, a light pen, a voice input device, a touch screen, a pedometer, or a gravity sensor, etc.

[0230] The output device may exemplarily include devices for allowing information to be output to the user, such as a display screen, a printer, a speaker, etc.

[0231] The processor executes the programs stored in the memory and calls other devices.

[0232] Those skilled in the art can understand that Figure 12 the structure shown in

[0233] is only a block diagram of some structures related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different component layout.

[0234] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, which when executed by a processor implements the steps in the above method embodiments.

[0235] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or fully authorized by all parties, and the collection, use, and processing of the relevant data need to comply with the relevant regulations.

[0236] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memories. Non-volatile memories can include read-only memory (ROM), magnetic tapes, floppy disks, flash memories, optical memories, high-density embedded non-volatile memories, resistive random access memories (ReRAM), magnetoresistive random access memories (MRAM), ferroelectric random access memories (FRAM), phase change memories (PCM), graphene memories, etc. Volatile memories can include random access memory (RAM) or external cache memories, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0237] The databases involved in the embodiments provided in the present application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., without limitation. The processors involved in the embodiments provided in the present application can be general-purpose processors, central processors, graphics processors, digital signal processors, programmable logics, data processing logics based on quantum computing, etc., without limitation.

[0238] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered to be within the scope described in this specification.

[0239] In this article, specific examples are used to elaborate on the principles and implementation manners of the present application. The descriptions of the above embodiments are only used to help understand the methods and core ideas of the present application; at the same time, for those of ordinary skill in the art, according to the ideas of the present application, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to the present application.

Claims

1. An industrial data identification method, characterized in that: The industrial data identification method comprises: Acquire display images of industrial equipment; Preprocessing the displayed image to obtain a preprocessed image; Performing image segmentation processing on the preprocessed image to obtain sub-images, and determining a target area containing character information from the sub-images; The target area is identified to obtain text data, and the text data is corrected to generate industrial data corresponding to the industrial equipment.

2. The industrial data identification method according to claim 1, characterized in that: Obtain display images of industrial equipment, including: When the device type of the industrial device is a multi-mode display industrial device, the display image is an image obtained by an industrial control program inside the industrial device performing a screenshot operation on a target page.

3. The industrial data identification method according to claim 2, characterized in that: The industrial control program inside the industrial device performs a screenshot operation on a target page of the industrial device, including: Each target page is displayed in the display area of ​​the industrial device and a screenshot operation is performed.

4. The industrial data identification method according to claim 3, characterized in that: Each target page is displayed in a display area of ​​the industrial device, including: At least one interactive operation on the page switching control is performed according to the page switching control position information in the page switching configuration file to switch from the current page to the next target page.

5. The industrial data identification method according to claim 1, characterized in that: Performing image segmentation processing on the preprocessed image to obtain a sub-image, and determining a target area containing character information from the sub-image, including: According to the image segmentation configuration file, the preprocessed image is subjected to image segmentation processing to obtain a sub-image containing character information; the background area in the sub-image is removed to obtain the target area containing character information; the image segmentation configuration file includes at least one of the following: area segmentation position information, area parameter information and area description information; the area parameter information includes parameter information in the target area, and the area description information includes information describing the target area; or The preprocessed image is input into a trained image segmentation model for image segmentation processing to obtain a target area containing character information output by the image segmentation model; the image segmentation model is trained based on historical display images and corresponding area annotation results.

6. The industrial data identification method according to claim 5, characterized in that: The image segmentation model is constructed by the following steps: Acquire a historical display image and divide the historical display image into a training set and a test set; the historical display image is annotated with the region annotation result; Using the initial model to segment the historical display images in the training set to obtain a regional prediction result; Calculate the loss function according to the region prediction result and the corresponding region labeling result, minimize the loss function, and use an iterative algorithm to adjust the model parameters of the initial model until the loss function is minimized or the number of iterations is reached, so as to obtain the image segmentation model to be tested; Inputting the historical display image in the test set into the image segmentation model to be tested to obtain the corresponding region segmentation result; Corresponding evaluation data is determined based on the region segmentation result and the region labeling result corresponding to the test set, and when the evaluation data meets a preset condition, the image segmentation model to be tested is determined to be the image segmentation model.

7. The industrial data identification method according to claim 1, characterized in that: Identifying the target area, obtaining text data, and performing correction processing on the text data to generate industrial data corresponding to the industrial equipment, including: Using an optical character recognition (OCR) algorithm to identify the target area and obtain text data; Determine whether there is a historical text corresponding to the text data; the historical text is data collected before the text data and obtained through recognition processing; When there is a historical text, the text data is compared with the historical text according to the data correction rule, and the industrial data corresponding to the target area is determined according to the comparison result.

8. The industrial data identification method according to claim 7, characterized in that: The data correction rules include: data format; According to the data correction rules, the text data is compared with the historical text, and the industrial data corresponding to the target area is determined according to the comparison result, including: Matching the text data with the data format of the historical text; If the matching fails, the text data is used as the text to be corrected; According to the data correction rules, the text to be corrected is corrected to obtain the industrial data corresponding to the target area.

9. The industrial data identification method according to claim 8, characterized in that: The data correction rules also include: at least one parameter type and the upper and lower fluctuation range of the data corresponding to each parameter type; Before or after matching the text data with the data format of the historical text, the method further includes: Obtaining the upper and lower fluctuation range of the text data relative to the historical text data as the target fluctuation range; According to the parameter type to which the text data belongs, the target fluctuation range of the text data is matched with the upper and lower fluctuation ranges of the data corresponding to the parameter type.

10. The industrial data identification method according to any one of claims 1 to 9, characterized in that: Obtain display images of industrial equipment, including: When the device type of the industrial device is a multi-mode display industrial device, the display image is an image obtained by the industrial control program inside the industrial device taking screenshots of all display pages of the industrial device and filtering non-target pages in all screenshots.

11. The industrial data identification method according to claim 1, characterized in that: When the device type of the industrial device is a single-mode display industrial device, the display image is an image obtained by an industrial camera performing an image acquisition operation on a display area of ​​the industrial device.

12. An industrial data identification device, characterized in that: The industrial data identification device comprises: An acquisition module, used for acquiring display images of industrial equipment; A preprocessing module, used for preprocessing the display image to obtain a preprocessed image; A segmentation module, used for performing image segmentation processing on the preprocessed image to obtain sub-images, and determining a target area containing character information from the sub-images; The correction module is used to identify the target area, obtain text data, and perform correction processing on the text data to generate industrial data corresponding to the industrial equipment.

13. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the industrial data identification method according to any one of claims 1 to 11.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the industrial data identification method described in any one of claims 1 to 11 are implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the industrial data identification method described in any one of claims 1 to 11 are implemented.

Images