A time watermark extraction method and device based on picture optical identification

Abstract

The application discloses a time watermark extraction method and device based on picture optical identification, in which an operation and maintenance personnel submits an equipment operation and maintenance clock-in picture and current latitude and longitude position information through a terminal. After being acquired by a work order system, the corresponding pre-stored equipment latitude and longitude position information is called, the actual distance between the two is calculated and compared with a set threshold value; if the distance is smaller than the threshold value, the picture is determined to be valid to enter a subsequent step, otherwise, the picture is determined to be invalid and a re-submission instruction is sent to the terminal. Through an open source large model, text information is extracted from the valid picture by optical identification, compared with a preset time format regular expression, and a time text segment existing in the comparison is intercepted, otherwise, the extraction is re-identified; the time segment is converted into a standard time object according to a set format, and a manually filled work order completion time in the work order system is replaced, so that automatic correction is realized. The application solves the problems of low efficiency and easy error of manual identification of the picture time watermark, and lack of a picture authenticity verification method.

Description

Technical Field

[0001] This invention relates to the field of computer data processing technology, and specifically to a method and apparatus for extracting time watermarks based on image optical recognition. Background Technology

[0002] With the widespread adoption of IoT technology, the large-scale deployment of IoT devices has placed higher demands on the professionalism and timeliness of device operation and maintenance management. Various IoT device operation and maintenance systems have established work order management systems as core management tools to assess the timeliness of equipment maintenance response and task completion by operation and maintenance personnel. In practical applications of work order management, the industry commonly uses photos of operation and maintenance personnel submitting equipment maintenance check-in at the site as core evidence of work order completion. The work order review end relies on these check-in photos to conduct manual verification of the timeliness of operation and maintenance. This method has become a standard implementation approach in the field of IoT device operation and maintenance work order management, adapting to the operation and maintenance management needs of IoT devices in various scenarios.

[0003] The existing technology that relies on maintenance personnel manually submitting check-in photos to verify work order timeliness has many technical defects and management loopholes: On the one hand, the operation of maintenance personnel submitting work order completion photos after on-site work often has a delay, which means that the manually filled-in work order completion time cannot truly reflect the actual response and work timeliness of equipment maintenance, making it difficult to achieve accurate assessment of maintenance work timeliness; on the other hand, the work order review end can only verify the actual work time by manually identifying the time watermark in the check-in photo. Manual identification is not only inefficient and greatly increases the workload of work order reviewers, but is also prone to identification errors due to human negligence. At the same time, there is a lack of effective means to verify the authenticity of check-in photos, and maintenance personnel may submit falsified non-on-site and non-real-time check-in photos, which further leads to the distortion of work order timeliness data and fails to provide accurate and reliable timeliness data support for IoT device maintenance management.

[0004] Therefore, there is an urgent need for a time watermark extraction method based on optical image recognition to solve the problems of low efficiency and error in manual identification of photo time watermarks, as well as the lack of means to verify the authenticity of photos. Summary of the Invention

[0005] To address these issues, this invention provides a time watermark extraction method and apparatus based on image optical recognition, which solves problems such as work order time distortion caused by delays in photo submission by maintenance personnel, low efficiency and error-proneness of manual photo time watermark recognition, and lack of photo authenticity verification methods, thereby achieving automatic correction of work order completion time.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a time watermark extraction method based on image optical recognition, characterized in that it includes:

[0007] Maintenance personnel submit photos of equipment maintenance check-in and their current latitude and longitude location information via the terminal;

[0008] After the work order system obtains the equipment maintenance check-in photo and the current latitude and longitude location information of the maintenance personnel, it retrieves the pre-stored corresponding equipment latitude and longitude location information and calculates the actual distance between the current latitude and longitude location information and the corresponding equipment latitude and longitude location information. The actual distance is compared with a set threshold. If the actual distance is less than the set threshold, the equipment maintenance check-in photo is determined to be a valid photo, and the next step is performed. If the actual distance is not less than the set threshold, the equipment maintenance check-in photo is determined to be an invalid photo, and a resubmission instruction is sent to the maintenance personnel's terminal.

[0009] The valid photos are subjected to optical image recognition using an open-source large model to extract text information from the valid photos.

[0010] The text information is compared with a preset time format regular expression to determine whether there is a time text segment that satisfies the time format regular expression; if it is determined to exist, the corresponding time text segment is extracted and the next step is performed; if it is determined not to exist, the image optical recognition and text information extraction processing is performed again.

[0011] Convert the time text fragment into a standard time object according to the set date format;

[0012] The original work order completion time manually entered by maintenance personnel in the work order system is replaced with the standard time object to achieve automatic correction of the work order completion time.

[0013] As a preferred embodiment of a time watermark extraction method based on image optical recognition, the formula for calculating the actual distance between the current latitude and longitude location information and the corresponding device latitude and longitude location information is as follows:

[0014]

[0015] In the formula, d is the actual distance; R is the average radius of the Earth; The current latitude of the operations and maintenance personnel; For the corresponding equipment latitude; The current longitude of the maintenance personnel; This corresponds to the longitude of the equipment.

[0016] As a preferred solution for time watermark extraction based on image optical recognition, in the process of performing image optical recognition on the effective photos using the open-source large model, the open-source large model is an open-source visual language AI model with the function of optical recognition of text in natural scene images, and adopts an end-to-end encoder-decoder dual-tower architecture; the encoder is a visual feature extraction tower that deeply integrates CNN and visual Transformer; the decoder is a Transformer text decoding tower based on a multi-head self-attention mechanism; the encoder includes an image input layer, a convolutional feature extraction layer, a block embedding layer, a ViT encoder layer, and a feature fusion output layer connected in sequence; the decoder includes a feature receiving layer, a multi-head self-attention layer, a cross-attention layer, a feedforward neural network layer, and a text output layer connected in sequence.

[0017] The open-source large model performs the following steps to identify the valid photos:

[0018] The effective photos are preprocessed sequentially by image grayscale conversion, noise removal, and tilt correction to obtain the processed photos;

[0019] The encoder extracts and fuses text feature vectors from the processed photo to generate text feature vectors.

[0020] The text feature vector is decoded and spliced ​​by the decoder to output the text information in the valid photo.

[0021] As a preferred embodiment of a time watermark extraction method based on image optical recognition, in the process of comparing the text information with the preset time format regular expression, the time format regular expression includes a combination format of year, month, day and hour, minute and second and a separate format of year, month, day;

[0022] The expression for the combined year, month, day, hour, minute, and second format is:

[0023] ^\d{4}[- / ]\d{1,2}[- / ]\d{1,2}\s*\d{1,2}[:·]\d{1,2}[:·]\d{1,2}

[0024] The expression for the year, month, and day in a separate format is:

[0025] ^\d{4}[- / ]\d{1,2}[- / ]\d{1,2}

[0026] In the formula, ^ represents the starting position of the matched time text; \d is the exclusive abbreviation for "number", matching any single digit from 0 to 9; {4} means that 4 consecutive characters must be matched; {1,2} means that 1 or 2 consecutive characters can be matched; [- / ] is a separator specifically adapted for dates; \s* matches 0 or more whitespace characters, adapting to spaces between dates and times; [:·] is a separator specifically adapted for hours, minutes, and seconds.

[0027] As a preferred embodiment of a time watermark extraction method based on image optical recognition, during the extraction of the corresponding time text segment, if there are several time text segments in the text information that satisfy the time format regular expression, then the time text segment with the largest timestamp value is selected as the target segment for conversion; the formula for calculating the timestamp value is:

[0028]

[0029] In the formula, T is the timestamp value; Y is the difference between the year in the time text fragment and the base year; M is the month in the time text fragment; D is the date in the time text fragment; h is the hour in the time text fragment; m is the minute in the time text fragment; and s is the second in the time text fragment.

[0030] As a preferred embodiment of a time watermark extraction method based on image optical recognition, the standard time object is a timestamp format or the YYYY-MM-DD hh:mm:ss format of the ISO8601 standard.

[0031] This invention also provides a time watermark extraction device based on image optical recognition, employing the above-mentioned time watermark extraction method based on image optical recognition, including:

[0032] The check-in photo set location information upload module is used by maintenance personnel to submit equipment maintenance check-in photos and current latitude and longitude location information through the terminal;

[0033] The latitude and longitude location information comparison module is used by the work order system to retrieve the equipment maintenance check-in photo and the current latitude and longitude location information of the maintenance personnel, retrieve the pre-stored corresponding equipment latitude and longitude location information, and calculate the actual distance between the current latitude and longitude location information and the corresponding equipment latitude and longitude location information; compare the actual distance with a set threshold; if the actual distance is less than the set threshold, the equipment maintenance check-in photo is determined to be a valid photo, and the next step is performed; if the actual distance is not less than the set threshold, the equipment maintenance check-in photo is determined to be an invalid photo, and a resubmission instruction is sent to the maintenance personnel's terminal;

[0034] The image text information extraction module is used to perform image optical recognition on the valid photos using an open-source large model to extract the text information in the valid photos.

[0035] The time text fragment acquisition module is used to compare the text information with a preset time format regular expression to determine whether there is a time text fragment that satisfies the time format regular expression; if it is determined to exist, the corresponding time text fragment is extracted and the next operation is performed; if it is determined not to exist, the image optical recognition and text information extraction processing is performed again.

[0036] The time text fragment conversion module is used to convert the time text fragment into a standard time object according to a set date format.

[0037] The automatic work order completion time correction module is used to replace the original work order completion time manually filled in by maintenance personnel in the work order system with the standard time object, thereby realizing the automatic correction of the work order completion time.

[0038] As a preferred embodiment of a time watermark extraction device based on image optical recognition, the formula for calculating the actual distance between the current latitude and longitude location information and the corresponding device latitude and longitude location information in the latitude and longitude location information comparison module is as follows:

[0039]

[0040] In the formula, d is the actual distance; R is the average radius of the Earth; The current latitude of the operations and maintenance personnel; For the corresponding equipment latitude; The current longitude of the maintenance personnel; This corresponds to the longitude of the equipment.

[0041] As a preferred embodiment of a time watermark extraction device based on image optical recognition, the text information extraction module in the photo, during the process of performing image optical recognition on the effective photo using the open-source large model, is an open-source visual language AI model with the function of optical recognition of text in natural scene images, and adopts an end-to-end encoder-decoder dual-tower architecture; the encoder is a visual feature extraction tower that deeply integrates CNN and visual Transformer; the decoder is a Transformer text decoding tower based on a multi-head self-attention mechanism; the encoder includes an image input layer, a convolutional feature extraction layer, a block embedding layer, a ViT encoder layer, and a feature fusion output layer connected in sequence; the decoder includes a feature receiving layer, a multi-head self-attention layer, a cross-attention layer, a feedforward neural network layer, and a text output layer connected in sequence;

[0042] The open-source large model performs the following steps to identify the valid photos:

[0043] The effective photos are preprocessed sequentially by image grayscale conversion, noise removal, and tilt correction to obtain the processed photos;

[0044] The encoder extracts and fuses text feature vectors from the processed photo to generate text feature vectors.

[0045] The text feature vector is decoded and spliced ​​by the decoder to output the text information in the valid photo.

[0046] As a preferred embodiment of a time watermark extraction device based on image optical recognition, in the time text fragment acquisition module, during the process of comparing the text information with the preset time format regular expression, the time format regular expression includes a combination format of year, month, day, hour, minute, and second, and a separate format of year, month, and day;

[0047] The expression for the combined year, month, day, hour, minute, and second format is:

[0048] ^\d{4}[- / ]\d{1,2}[- / ]\d{1,2}\s*\d{1,2}[:·]\d{1,2}[:·]\d{1,2}

[0049] The expression for the year, month, and day in a separate format is:

[0050] ^\d{4}[- / ]\d{1,2}[- / ]\d{1,2}

[0051] In the formula, ^ represents the starting position of the matched time text; \d is the exclusive abbreviation for "number", matching any single digit from 0 to 9; {4} means that 4 consecutive characters must be matched; {1,2} means that 1 or 2 consecutive characters can be matched; [- / ] is a separator specifically adapted for dates; \s* matches 0 or more whitespace characters, adapting to spaces between dates and times; [:·] is a separator specifically adapted for hours, minutes, and seconds.

[0052] As a preferred embodiment of a time watermark extraction device based on image optical recognition, in the time text fragment acquisition module, during the process of extracting the corresponding time text fragment, if there are several time text fragments in the text information that satisfy the time format regular expression, then the time text fragment with the largest timestamp value is selected as the target fragment for conversion; the formula for calculating the timestamp value is:

[0053]

[0054] In the formula, T is the timestamp value; Y is the difference between the year in the time text fragment and the base year; M is the month in the time text fragment; D is the date in the time text fragment; h is the hour in the time text fragment; m is the minute in the time text fragment; and s is the second in the time text fragment.

[0055] As a preferred embodiment of a time watermark extraction device based on image optical recognition, in the time text fragment conversion module, the standard time object is a timestamp format or the YYYY-MM-DD hh:mm:ss format of the ISO8601 standard.

[0056] The present invention has the following advantages:

[0057] First, it enables automated recognition and extraction of time watermarks from maintenance check-in photos and automatic correction of work order completion times, reducing manual intervention and improving the overall efficiency of work order time data processing. At the same time, it makes work order time data more consistent with actual maintenance operations, ensuring data objectivity.

[0058] Secondly, the authenticity of check-in photos is verified by using latitude and longitude location verification, which avoids the falsification of off-site and non-real-time photos from the source, making the basic data for work order timeliness assessment more authentic and reliable.

[0059] Third, it replaces the traditional model of manually identifying photo time watermarks and manually reviewing work order times, significantly reducing repetitive work for work order reviewers, shortening the entire work order review cycle, and improving the overall operational efficiency of maintenance work order management.

[0060] Fourth, relying on objective photo time watermarks and real on-site verification results to form the basis for performance evaluation of operation and maintenance efficiency can accurately reflect the actual response and operation status of operation and maintenance personnel, optimize the evaluation mechanism, and motivate operation and maintenance personnel to carry out on-site operations.

[0061] Fifth, the technical solution is tailored to the actual operation scenarios of IoT device maintenance, and the operation of each step is adapted to the on-site workflow of maintenance personnel. It is easy to implement and can ensure the standardization of equipment maintenance timeliness management, thus helping the IoT device maintenance system to operate efficiently. Attached Figure Description

[0062] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.

[0063] The structures, proportions, sizes, etc. illustrated in this specification are only for the purpose of assisting those skilled in the art in understanding and reading the content disclosed herein, and are not intended to limit the conditions under which the present invention can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.

[0064] Figure 1 This is a flowchart illustrating a time watermark extraction method based on image optical recognition provided in Embodiment 1 of the present invention;

[0065] Figure 2 This is a schematic diagram illustrating the specific implementation process of a time watermark extraction method based on image optical recognition provided in Embodiment 1 of the present invention;

[0066] Figure 3 This is a schematic diagram of the architecture of a time watermark extraction device based on image optical recognition provided in Embodiment 2 of the present invention. Detailed Implementation

[0067] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0068] Example 1

[0069] See Figure 1 and Figure 2 Embodiment 1 of the present invention provides a time watermark extraction based on image optical recognition, including the following steps:

[0070] S1. Maintenance personnel submit equipment maintenance check-in photos and current latitude and longitude location information through the terminal;

[0071] S2. After the work order system obtains the equipment maintenance check-in photo and the current latitude and longitude location information of the maintenance personnel, it retrieves the pre-stored corresponding equipment latitude and longitude location information and calculates the actual distance between the current latitude and longitude location information and the corresponding equipment latitude and longitude location information; it compares the actual distance with a set threshold. If the actual distance is less than the set threshold, the equipment maintenance check-in photo is determined to be a valid photo, and the next step is performed; if the actual distance is not less than the set threshold, the equipment maintenance check-in photo is determined to be an invalid photo, and a resubmission instruction is sent to the maintenance personnel's terminal.

[0072] S3. Perform image optical recognition on the valid photos using an open-source large model to extract text information from the valid photos;

[0073] S4. Compare the text information with a preset time format regular expression to determine whether there is a time text segment that satisfies the time format regular expression; if it is determined to exist, extract the corresponding time text segment and proceed to the next step; if it is determined not to exist, re-perform image optical recognition and text information extraction processing.

[0074] S5. Convert the time text fragment into a standard time object according to the set date format;

[0075] S6. Replace the original work order completion time manually filled in by the maintenance personnel in the work order system with the standard time object to realize the automatic correction of the work order completion time.

[0076] In this embodiment, in step S1, the maintenance personnel submit equipment maintenance check-in photos and current latitude and longitude location information through the terminal.

[0077] Specifically, the terminal is a mobile smart terminal equipped with an operation and maintenance management application. After completing on-site maintenance work on the IoT devices bound to the corresponding work order, the operation and maintenance personnel take a check-in photo through the terminal, which includes the device itself, the on-site environment, and the original time watermark of the camera. Simultaneously, location permissions must be enabled on the terminal, which simultaneously collects the current real-time latitude and longitude location information at the moment the photo is submitted. This ensures a strong correlation between the latitude and longitude location information and the act of taking the check-in photo and the location, preventing situations where photos are resubmitted afterward or the location is modified and submitted from a different location. The operation and maintenance personnel then submit the check-in photo and the simultaneously collected latitude and longitude location information to the back-end work order system through the terminal, completing the reporting of on-site work data.

[0078] In this embodiment, in step S2, after the work order system obtains the equipment maintenance check-in photo and the current latitude and longitude location information of the maintenance personnel, it retrieves the pre-stored corresponding equipment latitude and longitude location information and calculates the actual distance between the current latitude and longitude location information and the corresponding equipment latitude and longitude location information; it compares the actual distance with a set threshold. If the actual distance is less than the set threshold, the equipment maintenance check-in photo is determined to be a valid photo, and the next step is performed; if the actual distance is not less than the set threshold, the equipment maintenance check-in photo is determined to be an invalid photo, and a resubmission instruction is sent to the maintenance personnel's terminal.

[0079] Specifically, after receiving the check-in photo and latitude / longitude location information submitted by the terminal, the work order system retrieves the latitude / longitude location information of the equipment installation point, which is pre-stored in the system database and entered during equipment installation and deployment and cannot be arbitrarily modified, based on the unique identifier of the equipment bound to the current work order. Using the Earth's spherical distance calculation formula, the system calculates the actual straight-line distance between the current latitude / longitude location submitted by the maintenance personnel and the pre-stored latitude / longitude location of the equipment. The set threshold is 200 meters, which is used to limit the maintenance personnel to be within a reasonable range of the equipment site.

[0080] The formula for calculating the actual distance between the current latitude and longitude location information and the corresponding device latitude and longitude location information is as follows:

[0081]

[0082] In the formula, d is the actual distance; R is the average radius of the Earth; The current latitude of the operations and maintenance personnel; For the corresponding equipment latitude; The current longitude of the maintenance personnel; This corresponds to the longitude of the equipment.

[0083] After completing the distance calculation, the work order system executes the threshold comparison logic: if the actual distance is less than 200 meters, it is determined that the maintenance personnel have arrived at the equipment site, the check-in photo is marked as a valid photo, stored in the corresponding attachment library of the work order, and enters the subsequent identification process; if the actual distance is greater than or equal to 200 meters, it is determined that the maintenance personnel have not arrived at the equipment site, the check-in photo has the risk of being forged, it is marked as an invalid photo, and at the same time, a resubmission instruction containing the reason for invalidity is pushed to the maintenance personnel's terminal, the invalid submission log is recorded, and the maintenance personnel are guided to resubmit the on-site check-in, thus avoiding invalid entry of off-site photos from the source.

[0084] In this embodiment, in step S3, the effective photos are subjected to image optical recognition using an open-source large model to extract text information from the effective photos.

[0085] Specifically, the open-source large model is an open-source visual language AI model with optical text recognition capabilities for natural scene images, employing an end-to-end encoder-decoder dual-tower architecture. The encoder is a visual feature extraction tower that deeply integrates CNN and Visual Transformer; the decoder is a Transformer text decoding tower based on a multi-head self-attention mechanism. The encoder includes, in sequence, an image input layer, a convolutional feature extraction layer, a block embedding layer, a ViT encoder layer, and a feature fusion output layer; the decoder includes, in sequence, a feature receiving layer, a multi-head self-attention layer, a cross-attention layer, a feedforward neural network layer, and a text output layer.

[0086] The steps for performing optical image recognition on the valid photograph are as follows:

[0087] Standardized image preprocessing is performed on valid photos, including image grayscale conversion, Gaussian filtering for noise reduction, tilt angle correction, and contrast enhancement, to solve problems such as uneven lighting, image tilt, and blur noise in on-site photos, thereby improving the accuracy of text recognition.

[0088] The preprocessed photo is input into the open-source large model. The convolutional neural network layer of the model encoder extracts the low-level visual features such as the edge and texture of the text in the photo. The visual Transformer layer extracts the global semantic features of the text in the photo, and the multi-dimensional feature fusion is completed to generate the text feature vector.

[0089] The model decoder uses an attention mechanism to map visual features to text sequences, ultimately decoding and outputting all text information contained in valid photos. This includes all identifiable text content such as the original time watermark, device number, and on-site environmental text. The extracted text information is then bound and stored with the corresponding work order and valid photos, providing a data foundation for subsequent time-based filtering.

[0090] In this embodiment, in step S4, the text information is compared with a preset time format regular expression to determine whether there is a time text segment that satisfies the time format regular expression; if it is determined to exist, the corresponding time text segment is extracted and the next operation is performed; if it is determined not to exist, the image optical recognition and text information extraction processing is performed again.

[0091] Specifically, the preset time format regular expression is pre-adapted to the mainstream formats of native time watermarks in photos taken by mobile devices, including two types of matching rules: year-month-day + hour-minute-second combination format and year-month-day single format, which can cover the time expression forms of the native watermarks of most Android and iOS devices on the market.

[0092] The expression for the year-month-day-hour-minute-second combination format is as follows:

[0093] ^\d{4}[- / ]\d{1,2}[- / ]\d{1,2}\s*\d{1,2}[:·]\d{1,2}[:·]\d{1,2}

[0094] The expression for the year, month, and day in a separate format is:

[0095] ^\d{4}[- / ]\d{1,2}[- / ]\d{1,2}

[0096] In the formula, ^ represents the starting position of the matched time text; \d is the exclusive abbreviation for "number", matching any single digit from 0 to 9; {4} means that 4 consecutive characters must be matched; {1,2} means that 1 or 2 consecutive characters can be matched; [- / ] is a separator specifically adapted for dates; \s* matches 0 or more whitespace characters, adapting to spaces between dates and times; [:·] is a separator specifically adapted for hours, minutes, and seconds.

[0097] In this embodiment, all text information extracted in step S3 is traversed line by line and segment by segment. Each segment of text is fully matched and verified against a preset time format regular expression to determine whether there are character segments in the text that conform to the time format rules. If a matching time text segment is found, the corresponding time text segment is automatically selected as the target text, and the corresponding time text segment is truncated, deduplicated, and standardized by removing spaces before proceeding to the subsequent format conversion stage. If no matching time text segment is found, the system will re-perform optimized image preprocessing and optical recognition extraction operations on the valid photo, adjust the photo sharpening and contrast parameters, and then re-identify to avoid time extraction failure due to single recognition error. If multiple recognitions still fail to match a valid time segment, the photo will be marked as abnormal and pushed to the work order review end for manual intervention.

[0098] If the text information contains several time text segments that satisfy the time format regular expression, then the time text segment with the largest timestamp value is selected as the target segment for conversion; the formula for calculating the timestamp value is:

[0099]

[0100] In the formula, T is the timestamp value; Y is the difference between the year in the time text fragment and the base year; M is the month in the time text fragment; D is the date in the time text fragment; h is the hour in the time text fragment; m is the minute in the time text fragment; and s is the second in the time text fragment.

[0101] In this embodiment, in step S5, the time text fragment is converted into a standard time object according to the set date format.

[0102] Specifically, the time text fragment extracted in step S4 is decomposed into fields, parsing out the year, month, date, hour, minute, and second information contained in the text. Missing seconds and month / date / hour / minute / second fields with fewer than two digits are completed. According to the set date format, the decomposed time fields are converted into a standard time object that the system can directly read, compare, and calculate. After conversion, the system performs a validity check on the standard time object, including whether the year is within a reasonable business range, whether the month is between 1 and 12, whether the date matches the number of days in the corresponding month, and whether the time is between 00:00:00 and 23:59:59. If the check passes, the standard time object is locked; if the check fails, the process returns to step S4 to re-match and extract the data, ensuring that the final output time object has a uniform and valid format.

[0103] The standard time object is a timestamp format or the YYYY-MM-DD hh:mm:ss format of the ISO8601 standard.

[0104] In this embodiment, in step S6, the original work order completion time manually filled in by the maintenance personnel in the work order system is replaced with the standard time object to realize the automatic correction of the work order completion time.

[0105] Specifically, the original work order completion time field manually filled in by the maintenance personnel in the corresponding work order is retrieved, and the original time data is backed up in read-only format and stored in the work order audit log. The standard time object generated in step S5 is written into the actual completion time field of the work order to replace the original manually filled time. At the same time, the work order status is updated to "completed and awaiting settlement", and the corrected work order completion time is synchronized to the settlement system.

[0106] After the replacement operation is completed, the system will generate a complete work order correction record, including the original manually entered time, the corrected standard time, the corresponding check-in photo identifier, latitude and longitude verification results, time extraction log and other full-link information, and store it in the work order system's operation log library, which can be used by reviewers and managers to retrospectively query the entire process. The objectively extracted photo shooting time is used as the sole timeliness basis for the completion of the work order, realizing the automated and standardized correction of the work order completion time.

[0107] The application scenarios of this invention are as follows:

[0108] In the scenario of operation and maintenance management of smart municipal IoT equipment, this invention can connect to the municipal operation and maintenance work order management system to carry out timeliness management of operation and maintenance work orders of various terminals such as municipal street lights, smart manhole covers, and traffic monitoring equipment. By verifying the authenticity of on-site check-in photos and automatically extracting time watermarks, the work order completion time can be automatically corrected, adapting to the operation and maintenance management needs of municipal equipment that is widely distributed and scattered.

[0109] In the operation and maintenance management of production equipment and smart meters in industrial plants, this invention can be adapted to the full-process work order management of factory equipment inspection and fault repair. It uses the original time watermark of the on-site check-in photo as the core basis for work order completion, avoiding the situation of off-site check-in and time reporting discrepancies, and providing objective data support for the assessment of factory equipment operation and maintenance.

[0110] In the field operation and maintenance scenarios of telecommunications operators' base stations and transmission equipment, this invention can be adapted to the management characteristics of dispersed base station distribution and high proportion of field operations by maintenance personnel. It completes the screening of the authenticity of check-in photos by verifying the latitude and longitude location, and simultaneously completes the automatic extraction of photo time watermark and work order time correction, which greatly simplifies the entire process of work order review.

[0111] In the operation and maintenance scenarios of new energy infrastructure such as new energy charging piles and distributed photovoltaic power stations, this invention can uniformly manage the completion time of daily equipment inspection and fault repair work orders. Through automated time extraction and time correction, it reduces repetitive work of manual review and optimizes the assessment and management mechanism of operation and maintenance personnel.

[0112] In the context of intelligent security and fire protection equipment operation and maintenance management in industrial parks and commercial buildings, this invention can connect with the existing operation and maintenance work order system of the park to objectively verify the completion status of equipment inspection and fault repair work orders, reduce the manual workload of work order reviewers, and improve the overall operational efficiency of park equipment operation and maintenance management.

[0113] In the field operation and maintenance scenarios of power distribution network terminal equipment, this invention can adapt to the characteristics of wide distribution of power equipment and complex operation and maintenance site environment, complete the authenticity verification of operation and maintenance check-in photos and automatic extraction of time watermark, provide objective basis for the timeliness assessment of power operation and maintenance work orders, and ensure the standardized operation of the distribution network equipment operation and maintenance system.

[0114] It should be noted that the method of this disclosure embodiment can be executed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method of this disclosure embodiment, and the multiple devices will interact with each other to complete the method described.

[0115] It should be noted that the above description describes some embodiments of this disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0116] Example 2

[0117] See Figure 3 Embodiment 2 of the present invention also provides a time watermark extraction device based on image optical recognition, comprising:

[0118] The check-in photo set location information upload module 001 is used by maintenance personnel to submit equipment maintenance check-in photos and current latitude and longitude location information through the terminal.

[0119] The latitude and longitude location information comparison module 002 is used to retrieve the pre-stored corresponding equipment latitude and longitude location information after the work order system obtains the equipment maintenance check-in photo and the current latitude and longitude location information of the maintenance personnel, and calculates the actual distance between the current latitude and longitude location information and the corresponding equipment latitude and longitude location information; compares the actual distance with a set threshold; if the actual distance is less than the set threshold, the equipment maintenance check-in photo is determined to be a valid photo and the next step is performed; if the actual distance is not less than the set threshold, the equipment maintenance check-in photo is determined to be an invalid photo and a resubmission instruction is sent to the maintenance personnel's terminal.

[0120] The text information extraction module 003 in the photo is used to perform optical image recognition on the effective photo using an open-source large model to extract the text information in the effective photo.

[0121] The time text fragment acquisition module 004 is used to compare the text information with a preset time format regular expression to determine whether there is a time text fragment that satisfies the time format regular expression; if it is determined to exist, the corresponding time text fragment is extracted and the next operation is performed; if it is determined not to exist, the image optical recognition and text information extraction processing is performed again.

[0122] The time text fragment conversion module 005 is used to convert the time text fragment into a standard time object according to a set date format.

[0123] The work order completion time automatic correction module 006 is used to replace the original work order completion time manually filled in by the maintenance personnel in the work order system with the standard time object, so as to realize the automatic correction of the work order completion time.

[0124] In this embodiment, in the latitude and longitude location information comparison module 002, the formula for calculating the actual distance between the current latitude and longitude location information and the corresponding device latitude and longitude location information is as follows:

[0125]

[0126] In the formula, d is the actual distance; R is the average radius of the Earth; The current latitude of the operations and maintenance personnel; For the corresponding equipment latitude; The current longitude of the maintenance personnel; This corresponds to the longitude of the equipment.

[0127] In this embodiment, in the text information extraction module 003 of the photo, during the process of performing image optical recognition on the effective photo using the open-source large model, the open-source large model is an open-source visual language AI model with the function of optical recognition of text in natural scene images, and adopts an end-to-end encoder-decoder dual-tower architecture; the encoder is a visual feature extraction tower that deeply integrates CNN and visual Transformer; the decoder is a Transformer text decoding tower based on a multi-head self-attention mechanism; the encoder includes an image input layer, a convolutional feature extraction layer, a block embedding layer, a ViT encoder layer and a feature fusion output layer connected in sequence; the decoder includes a feature receiving layer, a multi-head self-attention layer, a cross-attention layer, a feedforward neural network layer and a text output layer connected in sequence.

[0128] The open-source large model performs the following steps to identify the valid photos:

[0129] The effective photos are preprocessed sequentially by image grayscale conversion, noise removal, and tilt correction to obtain the processed photos;

[0130] The encoder extracts and fuses text feature vectors from the processed photo to generate text feature vectors.

[0131] The text feature vector is decoded and spliced ​​by the decoder to output the text information in the valid photo.

[0132] In this embodiment, in the process of comparing the text information with the preset time format regular expression in the time text fragment acquisition module 004, the time format regular expression includes a combination format of year, month, day and time, minute and second and a separate format of year, month, day;

[0133] The expression for the combined year, month, day, hour, minute, and second format is:

[0134] ^\d{4}[- / ]\d{1,2}[- / ]\d{1,2}\s*\d{1,2}[:·]\d{1,2}[:·]\d{1,2}

[0135] The expression for the year, month, and day in a separate format is:

[0136] ^\d{4}[- / ]\d{1,2}[- / ]\d{1,2}

[0137] In the formula, ^ represents the starting position of the matched time text; \d is the exclusive abbreviation for "number", matching any single digit from 0 to 9; {4} means that 4 consecutive characters must be matched; {1,2} means that 1 or 2 consecutive characters can be matched; [- / ] is a separator specifically adapted for dates; \s* matches 0 or more whitespace characters, adapting to spaces between dates and times; [:·] is a separator specifically adapted for hours, minutes, and seconds.

[0138] In this embodiment, in the time text fragment acquisition module 004, during the process of extracting the corresponding time text fragment, if there are several time text fragments in the text information that satisfy the time format regular expression, then the time text fragment with the largest timestamp value is selected as the target fragment for conversion; the formula for calculating the timestamp value is:

[0139]

[0140] In the formula, T is the timestamp value; Y is the difference between the year in the time text fragment and the base year; M is the month in the time text fragment; D is the date in the time text fragment; h is the hour in the time text fragment; m is the minute in the time text fragment; and s is the second in the time text fragment.

[0141] In this embodiment, the standard time object in the time text fragment conversion module 005 is a timestamp format or the ISO8601 standard YYYY-MM-DD hh:mm:ss format.

[0142] It should be noted that the information interaction and execution process between the modules of the above system are based on the same concept as the method embodiment in Embodiment 1 of this application, and the resulting technical effects are the same as those in the method embodiment of this application. For details, please refer to the description in the method embodiment shown above in this application, and it will not be repeated here.

[0143] Example 3

[0144] Embodiment 3 of the present invention provides a non-transitory computer-readable storage medium storing program code for a time watermark extraction method based on image optical recognition. The program code includes instructions for executing the time watermark extraction method based on image optical recognition of Embodiment 1 or any possible implementation thereof.

[0145] Computer-readable storage media can be any available medium that a computer can access, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

[0146] Example 4

[0147] Embodiment 4 of the present invention provides an electronic device, including: a memory and a processor;

[0148] The processor and the memory communicate with each other via a bus; the memory stores program instructions that can be executed by the processor, and the processor can execute the time watermark extraction method based on image optical recognition according to Embodiment 1 or any possible implementation thereof.

[0149] Specifically, a processor can be implemented in hardware or software. When implemented in hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented in software, the processor can be a general-purpose processor that reads software code stored in memory. This memory can be integrated into the processor or located outside the processor and exist independently.

[0150] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable system. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0151] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing systems. They can be centralized on a single computing system or distributed across a network of multiple computing systems. Optionally, they can be implemented using program code executable by a computing system, thereby storing them in a storage system for execution by the computing system. In some cases, the steps shown or described can be performed in a different order than those presented herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0152] Although the present invention has been described in detail above with general descriptions and specific embodiments, modifications or improvements can be made to it, which will be obvious to those skilled in the art. Therefore, all such modifications or improvements made without departing from the spirit of the present invention fall within the scope of protection claimed by the present invention.

Claims

1. A time watermark extraction method based on image optical recognition, characterized in that, include: Maintenance personnel submit photos of equipment maintenance check-in and their current latitude and longitude location information via the terminal; After the work order system obtains the equipment maintenance check-in photo and the current latitude and longitude location information of the maintenance personnel, it retrieves the pre-stored corresponding equipment latitude and longitude location information and calculates the actual distance between the current latitude and longitude location information and the corresponding equipment latitude and longitude location information. The actual distance is compared with a set threshold. If the actual distance is less than the set threshold, the equipment maintenance check-in photo is determined to be a valid photo, and the next step is performed. If the actual distance is not less than the set threshold, the equipment maintenance check-in photo is determined to be an invalid photo, and a resubmission instruction is sent to the maintenance personnel's terminal. The valid photos are subjected to optical image recognition using an open-source large model to extract text information from the valid photos. The text information is compared with a preset time format regular expression to determine whether there is a time text fragment that satisfies the time format regular expression. If it is determined that it exists, the corresponding time text segment is extracted and the next operation is performed; If it is determined that it does not exist, then the image optical recognition and text information extraction process will be repeated. Convert the time text fragment into a standard time object according to the set date format; The original work order completion time manually entered by maintenance personnel in the work order system is replaced with the standard time object to achieve automatic correction of the work order completion time.

2. The time watermark extraction method based on image optical recognition according to claim 1, characterized in that, The formula for calculating the actual distance between the current latitude and longitude location information and the corresponding device latitude and longitude location information is as follows: In the formula, d is the actual distance; R is the average radius of the Earth; The current latitude of the operations and maintenance personnel; For the corresponding equipment latitude; The current longitude of the maintenance personnel; This corresponds to the longitude of the equipment.

3. The time watermark extraction method based on image optical recognition according to claim 2, characterized in that, In the process of performing optical image recognition on the valid photos using the open-source large model, the open-source large model is an open-source visual language AI model with the function of optical text recognition in natural scene images, and adopts an end-to-end encoder-decoder dual-tower architecture; the encoder is a visual feature extraction tower that deeply integrates CNN and visual Transformer; the decoder is a Transformer text decoding tower based on a multi-head self-attention mechanism; the encoder includes an image input layer, a convolutional feature extraction layer, a block embedding layer, a ViT encoder layer and a feature fusion output layer connected in sequence; the decoder includes a feature receiving layer, a multi-head self-attention layer, a cross-attention layer, a feedforward neural network layer and a text output layer connected in sequence; The open-source large model performs the following steps to identify the valid photos: The effective photos are preprocessed sequentially by image grayscale conversion, noise removal, and tilt correction to obtain the processed photos; The encoder extracts and fuses text feature vectors from the processed photo to generate text feature vectors. The text feature vector is decoded and spliced ​​by the decoder to output the text information in the valid photo.

4. The time watermark extraction method based on image optical recognition according to claim 3, characterized in that, In the process of comparing the text information with the preset time format regular expression, the time format regular expression includes a combination format of year, month, day, hour, minute, and second, and a separate format of year, month, and day; The expression for the combined year, month, day, hour, minute, and second format is: ^\d{4}[- / ]\d{1,2}[- / ]\d{1,2}\s*\d{1,2}[:·]\d{1,2}[:·]\d{1,2} The expression for the year, month, and day in a separate format is: ^\d{4}[- / ]\d{1,2}[- / ]\d{1,2} In the formula, ^ represents the starting position of the matched time text; \d is the exclusive abbreviation for "number", matching any single digit from 0 to 9; {4} means that 4 consecutive characters must be matched; {1,2} means that 1 or 2 consecutive characters can be matched; [- / ] is a separator specifically adapted for dates; \s* matches 0 or more whitespace characters, adapting to spaces between dates and times; [:·] is a separator specifically adapted for hours, minutes, and seconds.

5. The time watermark extraction method based on image optical recognition according to claim 4, characterized in that, During the extraction of the corresponding time-based text fragments, if the text information contains several time-based text fragments that satisfy the time format regular expression, then the time-based text fragment with the largest timestamp value is selected as the target fragment for conversion; the formula for calculating the timestamp value is: In the formula, T is the timestamp value; Y is the difference between the year in the time text fragment and the base year; M is the month in the time text fragment; D is the date in the time text fragment; h is the hour in the time text fragment; m is the minute in the time text fragment; and s is the second in the time text fragment.

6. The time watermark extraction method based on image optical recognition according to claim 5, characterized in that, The standard time object is in timestamp format or in the YYYY-MM-DD hh:mm:ss format of the ISO8601 standard.

7. A time watermark extraction device based on image optical recognition, employing the time watermark extraction method based on image optical recognition as described in any one of claims 1-6, characterized in that, include: The check-in photo set location information upload module is used by maintenance personnel to submit equipment maintenance check-in photos and current latitude and longitude location information through the terminal; The latitude and longitude location information comparison module is used by the work order system to retrieve the equipment maintenance check-in photo and the current latitude and longitude location information of the maintenance personnel, retrieve the pre-stored corresponding equipment latitude and longitude location information, and calculate the actual distance between the current latitude and longitude location information and the corresponding equipment latitude and longitude location information; compare the actual distance with a set threshold; if the actual distance is less than the set threshold, the equipment maintenance check-in photo is determined to be a valid photo, and the next step is performed; if the actual distance is not less than the set threshold, the equipment maintenance check-in photo is determined to be an invalid photo, and a resubmission instruction is sent to the maintenance personnel's terminal; The image text information extraction module is used to perform image optical recognition on the valid photos using an open-source large model to extract the text information in the valid photos. The time text fragment acquisition module is used to compare the text information with a preset time format regular expression to determine whether there is a time text fragment that satisfies the time format regular expression. If it is determined that it exists, the corresponding time text segment is extracted and the next operation is performed; If it is determined that it does not exist, then the image optical recognition and text information extraction process will be repeated. The time text fragment conversion module is used to convert the time text fragment into a standard time object according to a set date format. The automatic work order completion time correction module is used to replace the original work order completion time manually filled in by maintenance personnel in the work order system with the standard time object, thereby realizing the automatic correction of the work order completion time.

8. The time watermark extraction device based on image optical recognition according to claim 7, characterized in that, In the latitude and longitude location information comparison module, the formula for calculating the actual distance between the current latitude and longitude location information and the corresponding device latitude and longitude location information is as follows: In the formula, d is the actual distance; R is the average radius of the Earth; The current latitude of the operations and maintenance personnel; For the corresponding equipment latitude; The current longitude of the maintenance personnel; This corresponds to the longitude of the equipment.

9. A time watermark extraction device based on image optical recognition according to claim 8, characterized in that, In the image text information extraction module, during the process of performing image optical recognition on the effective image using the open-source large model, the open-source large model is an open-source visual language AI model with the function of optical recognition of text in natural scene images, and adopts an end-to-end encoder-decoder dual-tower architecture; the encoder is a visual feature extraction tower that deeply integrates CNN and visual Transformer; the decoder is a Transformer text decoding tower based on a multi-head self-attention mechanism; the encoder includes an image input layer, a convolutional feature extraction layer, a block embedding layer, a ViT encoder layer, and a feature fusion output layer connected in sequence; the decoder includes a feature receiving layer, a multi-head self-attention layer, a cross-attention layer, a feedforward neural network layer, and a text output layer connected in sequence; The open-source large model performs the following steps to identify the valid photos: The effective photos are preprocessed sequentially by image grayscale conversion, noise removal, and tilt correction to obtain the processed photos; The encoder extracts and fuses text feature vectors from the processed photo to generate text feature vectors. The text feature vector is decoded and spliced ​​by the decoder to output the text information in the valid photo.

10. A time watermark extraction device based on image optical recognition according to claim 9, characterized in that, In the time text fragment acquisition module, during the process of comparing the text information with the preset time format regular expression, the time format regular expression includes a combination format of year, month, day, hour, minute, and second, and a separate format of year, month, and day. The expression for the combined year, month, day, hour, minute, and second format is: ^\d{4}[- / ]\d{1,2}[- / ]\d{1,2}\s*\d{1,2}[:·]\d{1,2}[:·]\d{1,2} The expression for the year, month, and day in a separate format is: ^\d{4}[- / ]\d{1,2}[- / ]\d{1,2} In the formula, ^ indicates the starting position of the matched time text; \d is the exclusive abbreviation for "number", matching any single digit from 0 to 9; {4} means that 4 consecutive characters must be matched; {1,2} means that 1 or 2 consecutive characters can be matched; [- / ] is a separator specifically adapted for dates; \s* matches 0 or more whitespace characters, adapting to spaces between dates and times; [:·] is a separator specifically adapted for hours, minutes, and seconds. In the time text fragment acquisition module, during the process of extracting the corresponding time text fragment, if there are several time text fragments in the text information that satisfy the time format regular expression, then the time text fragment with the largest timestamp value is selected as the target fragment for conversion; the formula for calculating the timestamp value is: In the formula, T is the timestamp value; Y is the difference between the year in the time text segment and the base year; M is the month in the time text segment; D is the date in the time text segment; h is the hour in the time text segment; m is the minute in the time text segment; and s is the second in the time text segment. In the time-text fragment conversion module, the standard time object is a timestamp format or the ISO8601 standard YYYY-MM-DD hh:mm:ss format.

Images