Column number identification method and device

Abstract

Embodiments of the present application provide a column number identification method and device, which processes a pre-processed to-be-identified image through a column number detection model to obtain digital card position information, initial content information and confidence corresponding to the initial content information of the to-be-identified digital card, and determine a number identification result corresponding to the unit digit of the to-be-identified digital card, wherein the numbers in the to-be-identified digital card are used to represent the column number; based on the digital card position information and time sequence information corresponding to the to-be-identified image, the number identification result other than the unit digit of the to-be-identified digital card is determined, so as to obtain an initial column number identification result, and the initial column number identification result is verified based on auxiliary identification data, and the initial column number identification result successfully verified is determined as the column number identification result. The method effectively solves the problems of low recognition accuracy and poor robustness, and significantly improves the accuracy and robustness of recognizing the column number.

Description

Technical Field

[0001] This application relates to the field of image recognition technology, specifically to a method and apparatus for recognizing field numbers. Background Technology

[0002] In existing technologies, the method of field number recognition is to take a field image with a camera and use a deep learning algorithm to infer the field image to identify multiple key points of the field and thus determine the field.

[0003] However, the accuracy of recognition drops significantly in scenarios with uneven lighting, dirt obscuring the image, or significant differences in the material of the digital signs on the field. Furthermore, it is difficult to handle the problem of digital fragmentation caused by motion blur in dynamic video streams. As a result, the existing technology lacks multimodal data fusion capabilities for the recognition of field numbers, and cannot effectively verify the physical and rational nature of the recognition results. Consequently, the false detection rate is high, making it difficult to meet users' needs for real-time performance and robustness. Summary of the Invention

[0004] To address the problems in the existing technology, this application provides a method and apparatus for identifying field numbers, which can effectively solve the shortcomings of traditional technologies in identifying field numbers, such as low accuracy and poor robustness caused by uneven lighting, dirt occlusion, changes in the style of number plates, and environmental interference, and significantly improve the accuracy and robustness of identifying field numbers.

[0005] To solve at least one of the above problems, this application provides the following technical solution:

[0006] Firstly, this application provides a method for identifying field numbers, including:

[0007] The system receives video data and auxiliary recognition data, obtains multiple images to be recognized based on the video data, preprocesses each image to be recognized, and obtains preprocessed images to be recognized. The images to be recognized include digital signs to be recognized, the auxiliary recognition data includes thermal imaging information and depth data information, and the preprocessing includes channel conversion processing, illumination adaptive enhancement processing, and stain detection processing.

[0008] The pre-processed image to be identified is processed by the field number detection model to obtain the position information of the digital card, the initial content information and the confidence level corresponding to the initial content information. The initial content information with a confidence level exceeding the preset confidence threshold is determined as the number recognition result corresponding to the unit digit of the digital card to be identified. The numbers in the digital card to be identified are used to represent the field number.

[0009] The temporal information corresponding to the pre-processed images to be identified in multiple adjacent frames is determined. Based on the position information and temporal information of the digital card, the identification results of the number other than the last digit of the digital card to be identified are determined. The identification results of the number corresponding to the last digit and the identification results of the number other than the last digit are combined to obtain the initial column number identification result corresponding to the digital card to be identified. The initial column number identification result is verified based on auxiliary identification data. The successfully verified initial column number identification result is determined as the column number identification result.

[0010] Furthermore, it also includes: determining the illumination level corresponding to each image to be identified based on thermal imaging information and preset illumination levels, wherein the preset illumination levels include low illumination level, normal illumination level and high illumination level;

[0011] Histogram equalization is used to enhance the low-light-level regions in each image to be identified, and dynamic range compression is used to process the high-light-level regions in each image to be identified, resulting in a processed image to be identified.

[0012] If a shadow is detected in the area where the digital sign to be identified is located in the image to be identified after processing the lighting, the initial content information in the digital sign to be identified is completed by generative adversarial network.

[0013] Furthermore, it also includes: extracting the outline information of the digital sign to be identified by edge detection, and determining the digital sign to be identified as a digital sign to be identified that is covered by dirt when the outline information does not meet the preset connectivity conditions. The preset connectivity conditions include the existence of a broken outline line obtained based on the outline information and / or the outline line obtained based on the current outline information being different from the preset outline line.

[0014] By using context repair to process the digit sign that is obscured by stains, the repaired digit sign can be obtained.

[0015] Furthermore, it also includes: when the pattern of the digital card to be identified or the environment in which the digital card to be identified is changed, the digital card to be identified that was not successfully identified is identified as a difficult sample;

[0016] The updated field number detection model is obtained by updating the model parameters corresponding to the difficult samples.

[0017] Furthermore, it also includes: determining the material characteristics of the digital signs to be identified one by one through thermal imaging information, and identifying the digital signs whose material characteristics are different from the preset material characteristics as false detection digital signs;

[0018] Based on depth data information, the spatial arrangement order of the digital cards to be identified is determined, and the continuity between the spatial arrangement order and the initial column number identification results is compared to obtain the comparison results.

[0019] If the comparison result is successful and there are no false detections of the number plate, the current initial column number recognition result is determined to be successfully verified.

[0020] Furthermore, after obtaining the comparison results, it also includes:

[0021] If the comparison result is a failure and / or there is a false detection of the digital card, the current initial field number recognition result is determined to be a failure verification;

[0022] Remove the images to be identified corresponding to the falsely detected number signs, and repeat the steps of processing the pre-processed images to be identified through the column number detection model to obtain the number sign position information, initial content information and the confidence level corresponding to the initial content information of the number signs to be identified.

[0023] Furthermore, after determining the initial content information with a confidence level exceeding a preset confidence threshold as the identification result corresponding to the unit digit of the digital card to be identified, the process also includes:

[0024] When the difference between the recognition results of the units digit of adjacent number cards to be recognized is greater than 1, the motion trajectory of the number cards to be recognized in the pre-processed images of consecutive adjacent number cards to be recognized is determined based on the position information of the number cards.

[0025] Based on the motion trajectory, the missing identification results are completed by linear interpolation.

[0026] Secondly, this application provides a field number identification device, comprising:

[0027] The receiving module is used to receive video data and auxiliary recognition data, obtain multiple images to be recognized based on the video data, preprocess each image to be recognized to obtain a preprocessed image to be recognized, wherein the image to be recognized includes the digital sign to be recognized, the auxiliary recognition data includes thermal imaging information and depth data information, and the preprocessing includes channel conversion processing, illumination adaptive enhancement processing and stain detection processing.

[0028] The recognition module is used to process the pre-processed image to be recognized through the field number detection model to obtain the position information of the digital card, the initial content information and the confidence level corresponding to the initial content information. The initial content information with a confidence level exceeding the preset confidence threshold is determined as the number recognition result corresponding to the unit digit of the digital card to be recognized. The numbers in the digital card to be recognized are used to represent the field number.

[0029] The output module is used to determine the temporal information corresponding to the pre-processed images to be identified in multiple adjacent frames. Based on the position information and temporal information of the digital card, it determines the identification result of the number other than the last digit of the digital card to be identified. It combines the identification result of the number corresponding to the last digit and the identification result of the number other than the last digit to obtain the initial column number identification result corresponding to the digital card to be identified. It also verifies the initial column number identification result based on auxiliary identification data and determines the successfully verified initial column number identification result as the column number identification result.

[0030] Thirdly, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the field number recognition method.

[0031] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the field number identification method described above.

[0032] Fifthly, this application provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the field number identification method described above.

[0033] As can be seen from the above technical solution, this application provides a method and apparatus for identifying column numbers. It innovatively receives video data and auxiliary identification data, obtains multiple images to be identified based on the video data, preprocesses each image to be identified, and obtains preprocessed images to be identified. The images to be identified include a number plaque to be identified, and the auxiliary identification data includes thermal imaging information and depth data. The preprocessing includes channel conversion processing, adaptive illumination enhancement processing, and stain detection processing. The preprocessed images to be identified are processed by a column number detection model to obtain the position information, initial content information, and confidence level of the number plaque to be identified. Initial content information with a confidence level exceeding a preset confidence threshold is determined as the identification result corresponding to the last digit of the number plaque. Based on the temporal information corresponding to multiple adjacent preprocessed images to be identified, and based on the position information and temporal information of the number plaque, the identification results for numbers other than the last digit of the number plaque to be identified are determined, thereby obtaining the identification result corresponding to the number plaque to be identified. This method effectively addresses the shortcomings of uneven lighting, dirt occlusion, changes in the style of number signs, and environmental interference in the identification of column numbers, resulting in low recognition accuracy and poor robustness, and significantly improves the accuracy and robustness of column number identification. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 This is a flowchart illustrating the field number identification method in the embodiments of this application;

[0036] Figure 2 This is a structural diagram of the field number recognition device in the embodiments of this application;

[0037] Figure 3 This is a schematic diagram of the structure of the electronic device in the embodiments of this application.

[0038] Figure label:

[0039] Electronic device 9600, central processing unit 9100, memory 9140, communication module 9110, input unit 9120, audio processor 9130, display 9160, power supply 9170, buffer memory 9141, application / function storage unit 9142, data storage unit 9143, driver storage unit 9144, antenna 9111, speaker 9131, microphone 9132. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0041] The acquisition, storage, use, and processing of data in this application all comply with the relevant provisions of national laws and regulations.

[0042] To effectively address the shortcomings of traditional technologies in field number recognition, such as low accuracy and poor robustness caused by uneven lighting, dirt occlusion, variations in number plaque styles, and environmental interference, and to significantly improve the accuracy and robustness of field number recognition, this application provides an embodiment of a field number recognition method. See [link to embodiment]. Figure 1 The field number identification method specifically includes the following:

[0043] Step S101: Receive video data and auxiliary recognition data, obtain multiple images to be recognized based on the video data, preprocess each image to be recognized, and obtain the preprocessed image to be recognized.

[0044] The image to be identified includes the digital sign to be identified, the auxiliary identification data includes thermal imaging information and depth data information, and the preprocessing includes channel conversion processing, illumination adaptive enhancement processing and stain detection processing.

[0045] Optionally, this embodiment receives video data and auxiliary recognition data. Both video data and auxiliary recognition data come from multiple sensors. Video data can be acquired by a high-definition camera, and auxiliary recognition data can be acquired by a thermal imaging camera and a millimeter-wave radar. Auxiliary recognition data includes thermal imaging information and depth data information. The thermal imaging camera can acquire the data, and the depth data can be acquired by the millimeter-wave radar. In this way, video data and auxiliary recognition data can characterize the features and environmental information of the digital sign to be identified from different dimensions.

[0046] In addition, the received video data is decoded. This can be done by using the OpenCV library to perform frame extraction on the video stream at a preset frame extraction frequency (e.g., extracting one frame every two frames) to obtain a static image to be recognized corresponding to the video data.

[0047] In addition, each image to be identified undergoes preprocessing, which may include, but is not limited to, channel conversion, illumination-adaptive enhancement, and stain detection. Specifically:

[0048] Channel conversion processing converts the image to be recognized from the default channel format (e.g., BGR format) to the channel format required by the object detection algorithm (e.g., RGB format), so that the image to be recognized meets the input requirements of the deep learning model and avoids color information errors and recognition deviations caused by inconsistent channel order.

[0049] The illumination-adaptive enhancement process determines the temperature distribution of the current scene based on thermal imaging information, generates a corresponding temperature weight map, multiplies the temperature weight map pixel by pixel with the RGB image to obtain an image with fused temperature information, and then performs CLAHE (Adaptive Histogram Equalization) processing on the image. This process can suppress the loss of details in overexposed and shadowed areas, enhance the clarity and contrast of the image to be identified under different lighting conditions, and improve the visibility of digital signs.

[0050] Stain detection processing can be achieved by using a pre-trained stain segmentation sub-network to perform pixel-level segmentation of the image to be identified, identifying stain regions in the image and generating a stain mask, thereby determining the content information on the digital sign to be identified.

[0051] This embodiment achieves the acquisition of the image to be recognized from video data, preprocesses the image to be recognized, and obtains a preprocessed image to be recognized. This can steadily improve the quality and accuracy of the field number recognition results, reduce the interference of environmental factors on the recognition process, and improve usability.

[0052] Step S102: Process the pre-processed image to be identified through the field number detection model to obtain the position information of the digital card to be identified, the initial content information and the confidence level corresponding to the initial content information. The initial content information with a confidence level exceeding the preset confidence threshold is determined as the number recognition result corresponding to the unit digit of the digital card to be identified. The numbers in the digital card to be identified are used to represent the field number.

[0053] Optionally, in this embodiment, the pre-processed image to be identified is input into the field number detection model to obtain the digital card position information, initial content information, and confidence level corresponding to the initial content information of the digital card to be identified.

[0054] The field number detection model can be constructed based on the YOLOv8-OD network, with a CBAM attention module embedded after the backbone feature extraction layer to enhance the field number detection model's ability to focus on the features corresponding to the numbered cards to be identified. At the same time, a BiFPN multi-scale feature fusion structure can be adopted in the neck network of the field number detection model to improve the detection accuracy of the field number detection model for numbered cards of different sizes, ensuring that the position information (including coordinates, width, and height) of the numbered card to be identified in the image to be identified, as well as the initial content information (i.e., numeric characters) displayed on the numbered card, can be accurately identified. The confidence level of the initial content information indicates the credibility of the field number detection model for the recognition result.

[0055] Additionally, a confidence threshold, such as 0.8, can be preset to filter the initial content information and its corresponding confidence levels. Initial content information with a confidence level exceeding the preset confidence threshold is identified as the unit digit number of the number plate to be identified. For example, if the column number detection model identifies the number "5" displayed on the number plate to be identified, and its corresponding confidence level is 0.85, exceeding the preset confidence threshold of 0.8, then "5" is identified as the unit digit number of that number plate.

[0056] This embodiment achieves the determination of the position information, initial content information, and corresponding confidence level of the digital card to be identified through the field number detection model, and determines the numbering result corresponding to the last digit. It can maintain stable recognition performance, reduce confusion caused by recognition errors, and at the same time, by recognizing each digit of the field number separately, the computing power requirement of the field number detection model is reduced, thus lowering the cost.

[0057] Step S103: Determine the temporal information corresponding to the pre-processed images to be identified in multiple adjacent frames. Based on the position information and temporal information of the digital card, determine the identification results of the number other than the last digit of the digital card to be identified. Combine the identification results of the number corresponding to the last digit and the identification results of the number other than the last digit to obtain the initial column number identification result corresponding to the digital card to be identified. Verify the initial column number identification result based on the auxiliary identification data. Determine the successfully verified initial column number identification result as the column number identification result.

[0058] Optionally, this embodiment obtains the temporal information corresponding to multiple adjacent pre-processed images to be identified. The three-dimensional coordinates of the digital sign to be identified can be determined by depth data information, and the temporal information corresponding to the image to be identified can be determined by combining the camera's shooting timestamp.

[0059] In addition, based on the location and timing information of the number plate, the identification results of the number plate other than the last digit can be determined. Specifically, based on the displacement trajectory of the number plate in consecutive multi-frame images, the probability of the number plate appearing in positions other than the last digit (e.g., the tens and hundreds digits) can be predicted using a Kalman filter. The probability of appearance is then combined with the temperature gradient change in the thermal imaging information to perform joint probability calculation. Based on the spatial position change trend and temperature characteristics of the number plate, candidate results for the tens and hundreds digits are output.

[0060] In addition, the identification result corresponding to the determined unit digit is combined with the candidate results corresponding to the identification results of the tens digit, hundreds digit, etc., excluding the unit digit, to form the initial column number identification result corresponding to the number card to be identified.

[0061] In addition, the initial field number identification result is verified based on auxiliary identification data. For example, the initial field number identification result is compared with the universally unique identifier (UUID) of the digital sign in the depth data information. If the comparison matches, the verification is considered successful, and the initial field number identification result is determined as the field number identification result.

[0062] Furthermore, if the comparison is inconsistent, the initial field number recognition result can be corrected again by determining the distance between the digital sign to be identified and the camera, combined with thermal imaging information. If the corrected initial field number recognition result matches the UUID in the depth data information, then the corrected initial field number recognition result is determined as the field number recognition result.

[0063] Furthermore, before receiving video data and auxiliary recognition data, a perforated number plate can be installed on the field. The perforated number plate is a metal plate with the number part cut out.

[0064] This embodiment achieves accurate determination of the column number recognition result of the digital card to be identified by fusing time sequence information and auxiliary recognition data. It solves the problem of recognizing multiple digits in the number of the digital card to be identified, avoids column recognition errors or statistical errors caused by incomplete numbering, and improves accuracy and completeness.

[0065] This embodiment achieves accurate, complete, and reliable identification of field numbers by receiving video data and auxiliary recognition data, preprocessing the image to be identified, and using a field number detection model and a multi-dimensional data fusion and verification mechanism. This reduces the failure or error of field number identification caused by environmental factors and improves usability, accuracy, and robustness.

[0066] In some embodiments, each image to be identified is preprocessed to obtain a preprocessed image to be identified, including:

[0067] Based on thermal imaging information and preset illumination levels, the illumination level corresponding to each image to be identified is determined. The preset illumination levels include low illumination level, normal illumination level and high illumination level.

[0068] Histogram equalization is used to enhance the low-light-level regions in each image to be identified, and dynamic range compression is used to process the high-light-level regions in each image to be identified, resulting in a processed image to be identified.

[0069] If a shadow is detected in the area where the digital sign to be identified is located in the image to be identified after processing the lighting, the initial content information in the digital sign to be identified is completed by generative adversarial network.

[0070] Optionally, this embodiment can determine the temperature matrix of each image to be identified by the thermal imaging information of each image to be identified, determine the difference between the ambient temperature of each pixel in the image to be identified and the temperature of the digital sign to be identified, and determine the illumination level corresponding to each image to be identified according to the preset illumination level.

[0071] Specifically, temperature differences can be classified into two categories using a deep learning model (such as the lightweight MobileNetV3-Small). The preset illumination levels include low illumination, normal illumination, and high illumination. Low illumination indicates areas with small temperature differences, which may be in the shade or at night. Normal illumination indicates areas with moderate temperature differences. High illumination indicates areas with large temperature differences, which may be areas with strong direct sunlight.

[0072] The system performs binary classification based on preset illumination information and temperature differences in the image to be identified, in order to classify the illumination levels in the image to be identified.

[0073] Furthermore, the preset threshold for illumination level can be dynamically updated based on historical data to avoid misjudgments caused by fixed thresholds. The historical data can be the distribution of illumination intensity at different times of the day.

[0074] In addition, low-light areas are enhanced, high-light areas are compressed, and the existing state and data of normal-light areas are maintained to obtain the processed lighting data to be identified.

[0075] Among them, the enhancement of low-light areas can be achieved by limiting contrast adaptive histogram equalization (CLAHE), which divides the low-light areas of the image to be identified into a grid of a preset low-light processing size, such as an 8×8 grid. The histogram is calculated independently for each grid and the contrast amplification factor is limited (e.g., 2.0 times) to avoid excessive amplification of noise.

[0076] Furthermore, gamma correction (e.g., γ = 0.6) can be applied to the low-pixel value areas of the digital sign to be identified to improve shadow details.

[0077] Highlight region compression can be achieved based on Retinex theory. The high-light region in the image to be identified is decomposed into illumination component and reflection component. Logarithmic compression is performed on the illumination component (e.g., log compression coefficient = 0.8) while preserving the texture details of the reflection component.

[0078] Furthermore, color restoration can be performed on the compressed image to prevent color distortion caused by reflections from the digital sign to be identified.

[0079] Furthermore, the CLAHE and Retinex algorithms can be processed in parallel using OpenCV's CUDA module, enabling real-time processing of the images to be recognized on an NVIDIA Jetson Xavier. The gamma value and log compression coefficient can also be dynamically adjusted based on temperature differences in thermal imaging information; for example, when the temperature difference is >15℃, the log compression coefficient is reduced to 0.6 to reduce glare from the digital signage to be recognized.

[0080] Furthermore, when a shadow is detected in the area where the digital sign to be identified is located in the processed lighting data, the initial content information in the digital sign to be identified can be completed by a generative adversarial network. The shadow detection can be achieved by segmenting the shadow area using a lightweight U-Net model. The input of the lightweight U-Net model is the image to be identified and the thermal imaging temperature difference map determined according to the corresponding thermal imaging information, and the output is a shadow mask.

[0081] The area of ​​the digital sign to be identified is further located within the shadow mask. Specifically, if the intersection-union ratio (IoU) between the YOLOv8 detection box and the mask is greater than a preset IoU threshold, it can be determined that the area of ​​the digital sign to be identified is occluded by a shadow, for example, if the IoU is greater than 0.7.

[0082] The generative adversarial network (GAN) can be cGAN, the generator is a Transformer-based SwinIR architecture, and the discriminator is PatchGAN. The input of the GAN is the shadow region in the image to be recognized and the corresponding thermal imaging information of the shadow region. The output is the completed image patch of the digital card to be recognized. The loss function of the GAN can be, but is not limited to, L1 loss function, perceptual loss function and adversarial loss function. L1 loss function is used to ensure pixel-level accuracy, perceptual loss function is used to ensure the authenticity of the completed image patch of the digital card to be recognized, and adversarial loss function is used to ensure texture authenticity.

[0083] This embodiment implements illumination processing on the image to be identified, improves the processing quality of the image under complex lighting and shadow conditions, and enhances the accuracy of field number recognition.

[0084] In some embodiments, each image to be identified is preprocessed to obtain a preprocessed image to be identified, including:

[0085] The outline information of the digital sign to be identified is extracted by edge detection. If the outline information does not meet the preset connectivity conditions, the digital sign to be identified is identified as a digital sign to be identified that is covered by dirt. The preset connectivity conditions include the existence of a broken outline line obtained based on the outline information and / or the outline line obtained based on the current outline information is different from the preset outline line.

[0086] By using context repair to process the digit sign that is obscured by stains, the repaired digit sign can be obtained.

[0087] Optionally, this embodiment can use an edge detection algorithm (such as the Sobel operator or the Canny operator) to process the image to be identified and extract the contour information of the digital sign to be identified. The edge detection algorithm can detect significant changes in pixel intensity in the image to be identified, thereby locating the boundary of the digital sign to be identified.

[0088] Based on the edge detection results, the contour information of the digital sign to be identified is extracted. The contour information is used to represent the outer boundary of the digital sign to be identified. The contour line of the digital sign to be identified can be determined based on the contour information. If the contour information does not meet the preset connectivity conditions, the digital sign to be identified is identified as a digital sign to be identified that is covered by stains. The preset connectivity conditions include the existence of a broken contour line obtained based on the contour information and / or the contour line obtained based on the current contour information being different from the preset contour line.

[0089] One step is to check whether the extracted contour lines are broken. If the contour lines are discontinuous, it may be because dirt has covered part of the digital sign to be identified, causing the edge detection to be unable to extract the contour completely.

[0090] The outline is compared with the preset outline. The extracted outline is compared with the preset outline corresponding to the number sign. If the extracted outline is significantly different from the preset outline (e.g., shape mismatch, excessive difference in outline size, etc.), it can be determined that the number sign to be identified may be obscured by dirt. The preset outline is defined in advance based on the shape of a normal number sign to be identified.

[0091] In addition, for digital signs that are identified as being obscured by stains, context inpainting technology can be used to process them and obtain repaired digital signs. Context inpainting is an image restoration method that uses information from the unobscured areas of an image, combined with the image's context (such as texture, color, shape, etc.), to reconstruct the obscured parts.

[0092] Among them, texture analysis and reconstruction are used to analyze the texture features of the unobstructed area around the digital sign to be identified, and reconstruct the texture of the area obscured by the stain based on these texture features.

[0093] Color interpolation is used to calculate the color value of the stained area based on the color information of the area surrounding the digital sign to be identified, using an interpolation algorithm, so that the repaired area is consistent with the surrounding area in color.

[0094] Shape constraints are used to constrain the repaired outline by combining it with the preset outline of the digit sign to be identified, so that the outline of the repaired digit sign to be identified conforms to the preset outline.

[0095] Furthermore, after the repair is completed, the repaired digit sign can be evaluated to check whether the repaired area blends naturally with the surrounding area visually, while maintaining the integrity of the shape and content of the digit sign.

[0096] Furthermore, edge detection algorithms suitable for the digits to be identified, such as the Canny operator, can be used. The Canny operator can effectively balance edge detection accuracy and noise resistance, making it suitable for edge extraction of digits in complex backgrounds. The parameters of the edge detection algorithm (e.g., threshold, gradient direction) can also be adjusted based on the characteristics of the digit image (e.g., resolution, contrast) to improve edge detection accuracy.

[0097] During the context restoration process, deep learning models, such as convolutional neural networks (CNNs) or generative adversarial networks (GANs), can be used to generate more natural and accurate restored digital signs to be identified. Multi-scale restoration methods can also be used to analyze and restore the occluded areas of stains at different scales to improve the robustness and detail representation of the restoration.

[0098] This embodiment enables more accurate identification of digital signs during subsequent processing by repairing the signs to be identified. It maintains stable performance even in complex situations such as stains and obstructions, improving user experience, identification accuracy, and robustness.

[0099] In some embodiments, it also includes:

[0100] When the pattern of the digital card to be identified or the environment in which the digital card to be identified is changed, the digital card to be identified that was not successfully identified is identified as a difficult sample.

[0101] The updated field number detection model is obtained by updating the model parameters corresponding to the difficult samples.

[0102] Optionally, this embodiment also includes a dynamic adaptation mechanism to handle and optimize the recognition capability when faced with changes in the style or environment of the digital card to be recognized. That is, when changes are detected in the style of the digital card to be recognized (e.g., the shape, color, font, etc.) or the environment in which the digital card to be recognized is located (e.g., lighting conditions, background interference, etc.), the unrecognized digital card to be recognized is automatically identified and marked as a difficult sample. The difficult samples can be stored separately and used in the field number detection model update process.

[0103] In addition, based on the image data to be identified that is marked as difficult samples, the update mechanism of the model parameters corresponding to the field number detection model is triggered. That is, difficult samples are included in the training dataset, and the model parameters of the field number detection model are retrained or fine-tuned to obtain an updated field number detection model. The dynamic update mechanism enables the field number detection model to better adapt to the style and environmental changes of new digital signs to be identified, thereby improving the generalization ability and recognition accuracy of the field number detection model.

[0104] Furthermore, during the process of obtaining the digital card to be identified, the confidence level of the digital card in the identification result can be monitored in real time. When the confidence level of the digital card is lower than the preset confidence level threshold, or when the identification result is inconsistent with the identification result of adjacent frames, the digital card to be identified is marked as a difficult sample. Inconsistency is, for example, when three different identification results appear in five consecutive frames.

[0105] It can also assess the environment of the digital sign to be identified in real time through thermal imaging information and depth data. If the light intensity change exceeds the preset intensity change threshold (e.g., 20%) or the background interference (e.g., dust, fog) exceeds the preset interference threshold, it can also trigger the marking of difficult samples.

[0106] Furthermore, an online incremental learning algorithm can be used to activate the update mechanism of the field number detection model when a certain number of difficult samples (e.g., 100) are accumulated. During the update process, the difficult samples are mixed with the existing training data to retrain a portion of the parameters of the field number detection model without retraining the entire model, thus saving computational resources.

[0107] Furthermore, pseudo-labels can be generated through self-supervised learning for unlabeled hard samples. For example, time-series information (such as changes in the position and shape of the number card to be identified in consecutive adjacent images) can be used to infer the true content of the number card, and the pseudo-labels can be used to update the field number detection model.

[0108] This embodiment uses a difficult sample processing and model update mechanism to enable the column number detection model to quickly adapt and optimize recognition performance when faced with changes in the style of the digital card to be recognized or changes in the environment, thereby improving recognition accuracy, real-time performance and robustness.

[0109] In some embodiments, verifying the initial field number recognition result based on auxiliary recognition data includes:

[0110] The material characteristics of the digital signs to be identified are determined one by one by thermal imaging information. Digital signs whose material characteristics are different from the preset material characteristics are identified as falsely detected digital signs.

[0111] Based on depth data information, the spatial arrangement order of the digital cards to be identified is determined, and the continuity between the spatial arrangement order and the initial column number identification results is compared to obtain the comparison results.

[0112] If the comparison result is successful and there are no false detections of the number plate, the current initial column number recognition result is determined to be successfully verified.

[0113] Optionally, in this embodiment, the material characteristics of the digital sign to be identified are determined one by one using thermal imaging information. Since different materials have different temperature distributions under the same environment, the material characteristics of the digital sign to be identified can be determined using thermal imaging information.

[0114] For each digital sign to be identified, the similarity between its thermal imaging information and the thermal imaging information corresponding to the preset material characteristics of digital signs can be calculated to determine whether the material characteristics of the digital sign to be identified are the same as those of the preset material characteristics. When the similarity between the material characteristics of the digital sign to be identified and the preset material characteristics is lower than the preset material characteristic threshold, it is marked as a false detection digital sign and excluded from the subsequent verification process.

[0115] In addition, the depth information corresponding to each digital card to be identified is determined based on the depth data information, the spatial position of each digital card to be identified is determined, and they are arranged in order from left to right (or from top to bottom) to obtain the actual spatial arrangement order of the digital cards to be identified.

[0116] In addition, the spatial arrangement order is compared with the initial field number recognition result to check whether the numbering is continuous. For example, if the spatial arrangement order is "1, 2, 3, 4" and the initial field number recognition result is "1, 4, 2, 3", the comparison is judged to be unsuccessful.

[0117] Furthermore, successful verification is defined as the spatial arrangement order being consistent with the initial field number recognition result and the absence of falsely detected number cards. The field number recognition result can be determined based on the initial field number recognition result.

[0118] This embodiment verifies the initial comparison field number recognition result using thermal imaging information and depth data information, which improves the recognition accuracy of the digital card to be identified in complex environments. At the same time, it reduces misjudgments caused by single modality data and improves robustness.

[0119] In some embodiments, after obtaining the comparison results, the method further includes:

[0120] If the comparison result is a failure and / or there is a false detection of the digital card, the current initial field number recognition result is determined to be a failure verification;

[0121] Remove the images to be identified corresponding to the falsely detected number signs, and repeat the steps of processing the pre-processed images to be identified through the column number detection model to obtain the number sign position information, initial content information and the confidence level corresponding to the initial content information of the number signs to be identified.

[0122] Optionally, in this embodiment, if the comparison result is a failure or a false detection of a number plate, or if the comparison fails and a false detection of a number plate exists, the current initial field number identification result is determined to be a failure verification.

[0123] Among them, a comparison failure means that the spatial arrangement order is inconsistent with the initial field number recognition result. For example, the spatial arrangement order is "1, 2, 4, 3", while the initial field number recognition result is "1, 2, 3, 4", which is judged as a comparison failure.

[0124] Remove the images corresponding to falsely detected digit signs from the pre-processed images to be recognized, so that the images to be recognized after reprocessing are reliable and avoid the impact of falsely detected digit signs on the recognition results.

[0125] In addition, the pre-processed image to be identified after removing falsely detected number plates is re-input into the field number detection model for further processing to obtain the number plate location information, initial content information, and the confidence level corresponding to the initial content information. The reprocessed results are then verified again to ensure the accuracy and reliability of the field number recognition results.

[0126] Furthermore, when a false detection number is detected, false detection information can be fed back in real time, and the characteristics of the false detection number can be recorded for optimization of the field number detection model.

[0127] Graph theory algorithms can be used to optimize the spatial arrangement of the number cards to be identified, thereby improving the accuracy of the spatial arrangement. When a false positive number card is detected, an online incremental learning mechanism for the field number detection model can be activated to add the false positive sample to the training dataset, update the model parameters of the field number detection model, and improve the adaptability of the field number detection model to new situations. Furthermore, the parameters of the field number recognition algorithm can be automatically adjusted based on the false positive number cards, such as adjusting the confidence threshold and optimizing the model structure, to improve adaptability and robustness.

[0128] This embodiment enables verification using thermal imaging information and depth data, and allows for reprocessing in cases of comparison failure or false detection of digital signs, thereby improving the accuracy and robustness of the column number recognition results.

[0129] In some embodiments, after determining the initial content information with a confidence level exceeding a preset confidence threshold as the identification result of the unit digit of the digital card to be identified, the method further includes:

[0130] When the difference between the recognition results of the units digit of adjacent number cards to be recognized is greater than 1, the motion trajectory of the number cards to be recognized in the pre-processed images of consecutive adjacent number cards to be recognized is determined based on the position information of the number cards.

[0131] Based on the motion trajectory, the missing identification results are completed by linear interpolation.

[0132] Optionally, in this embodiment, after determining the identification result of the last digit of each number card to be identified, the difference between the identification results of adjacent number cards is checked. If the difference between the identification results of the last digit of adjacent number cards to be identified is greater than 1, that is, for example, the identification result jumps from "3" to "5", it is determined that there is a missing number.

[0133] In addition, based on depth data and digital sign location information, the spatial coordinates of the digital signs to be identified in consecutively adjacent pre-processed images to be identified are determined.

[0134] Among them, the motion trajectory of the digital card to be identified in consecutive adjacent images can be tracked by using a Kalman filter or optical flow algorithm. The motion trajectory is used to reflect the actual movement path of the digital card to be identified.

[0135] In addition, based on the determined motion trajectory, the position of the missing number is determined by linear interpolation. That is, based on the recognition results of the identified field numbers and the position information of the number card to be identified, a reasonable value of the missing number is calculated by linear interpolation, and the completed field number is inserted into the missing position to form a complete field number recognition result.

[0136] Furthermore, combining thermal imaging and depth data can improve the accuracy of motion trajectory tracking. For example, thermal imaging can be used to help determine the temperature changes of the digital sign to be identified, further confirming its location. Deep learning algorithms (such as LSTM or Transformer) can also be used to model and predict the motion trajectory of the digital sign, improving the accuracy and robustness of motion trajectory tracking.

[0137] Furthermore, the interpolation method can be dynamically adjusted based on the speed and acceleration of the digital sign to be identified. For example, when the digital sign to be identified moves at a relatively fast speed, a higher-order interpolation method can be used to improve the completion accuracy.

[0138] This embodiment realizes the completion of the number card to be identified by motion trajectory, which improves the completeness and robustness of the number recognition result, reduces the need for manual intervention, and improves the overall operating efficiency.

[0139] To effectively address the shortcomings of traditional technologies in field number recognition, such as low accuracy and poor robustness caused by uneven lighting, dirt occlusion, variations in number plaque styles, and environmental interference, and to significantly improve the accuracy and robustness of field number recognition, this application provides an embodiment of a field number recognition device for implementing all or part of the aforementioned field number recognition. See [link to embodiment]. Figure 2 The field number identification device specifically includes the following components:

[0140] The receiving module 10 is used to receive video data and auxiliary recognition data, obtain multiple images to be recognized based on the video data, preprocess each image to be recognized, and obtain a preprocessed image to be recognized. The image to be recognized includes a digital sign to be recognized, the auxiliary recognition data includes thermal imaging information and depth data information, and the preprocessing includes channel conversion processing, illumination adaptive enhancement processing and stain detection processing.

[0141] The recognition module 20 is used to process the pre-processed image to be recognized through the field number detection model to obtain the position information of the digital card to be recognized, the initial content information and the confidence level corresponding to the initial content information, and to determine the initial content information with a confidence level exceeding the preset confidence threshold as the number recognition result corresponding to the unit digit of the digital card to be recognized. The numbers in the digital card to be recognized are used to represent the field number.

[0142] The output module 30 is used to determine the temporal information corresponding to the pre-processed images to be identified in multiple adjacent frames. Based on the position information and temporal information of the digital card, it determines the identification result of the number other than the last digit of the digital card to be identified. It combines the identification result of the number corresponding to the last digit and the identification result of the number other than the last digit to obtain the initial column number identification result corresponding to the digital card to be identified. It verifies the initial column number identification result based on auxiliary identification data and determines the successfully verified initial column number identification result as the column number identification result.

[0143] As described above, the column number recognition device provided in this application embodiment can innovatively receive video data and auxiliary recognition data, obtain multiple images to be recognized based on the video data, preprocess each image to be recognized to obtain a preprocessed image to be recognized, wherein the image to be recognized includes a number sign to be recognized, the auxiliary recognition data includes thermal imaging information and depth data information, the preprocessing includes channel conversion processing, illumination adaptive enhancement processing and stain detection processing, and the preprocessed image to be recognized is processed by the column number detection model to obtain the position information of the number sign, the initial content information and the confidence level corresponding to the initial content information of the number sign to be recognized, and the initial content information with a confidence level exceeding a preset confidence threshold is determined as the number recognition result corresponding to the last digit of the number sign to be recognized, and the number recognition result other than the last digit of the number sign to be recognized is determined based on the temporal information corresponding to the preprocessed images of adjacent frames, the position information of the number sign and the temporal information, thereby obtaining the number recognition result corresponding to the number sign to be recognized. This method effectively addresses the shortcomings of uneven lighting, dirt occlusion, changes in the style of number signs, and environmental interference in the identification of column numbers, resulting in low recognition accuracy and poor robustness, and significantly improves the accuracy and robustness of column number identification.

[0144] From a hardware perspective, in order to effectively address the shortcomings of traditional technologies in identifying column numbers, such as low accuracy and poor robustness caused by uneven lighting, dirt occlusion, changes in the style of the number plaques, and environmental interference, and to significantly improve the accuracy and robustness of column number identification, this application provides an embodiment of an electronic device for implementing all or part of the column number identification method. The electronic device specifically includes the following components:

[0145] The system comprises a processor, memory, a communications interface, and a bus; wherein the processor, memory, and communications interface communicate with each other via the bus; the communications interface is used to realize information transmission between the field number recognition device and core business systems, user terminals, and related databases and other related devices; the logic controller can be a desktop computer, tablet computer, or mobile terminal, etc., and this embodiment is not limited to these. In this embodiment, the logic controller can be implemented with reference to the embodiments of the field number recognition method and the field number recognition device in the embodiments, the content of which is incorporated herein, and repeated details will not be described again.

[0146] It is understood that the user terminal may include smartphones, tablet computers, network set-top boxes, portable computers, desktop computers, personal digital assistants (PDAs), in-vehicle devices, smart wearable devices, etc. Among these, the smart wearable devices may include smart glasses, smartwatches, smart bracelets, etc.

[0147] In practical applications, the field number recognition method can be partially executed on the electronic device side as described above, or all operations can be completed in the client device. The choice can be made based on the processing power of the client device and the limitations of the user's usage scenario. This application does not impose any limitations on this. If all operations are completed in the client device, the client device may further include a processor.

[0148] The aforementioned client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission. The server may include a server on the task scheduling center side; in other implementation scenarios, it may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, a server cluster consisting of multiple servers, or a distributed server structure.

[0149] Figure 3 This is a schematic block diagram illustrating the system configuration of the electronic device 9600 according to an embodiment of this application. Figure 3 As shown, the electronic device 9600 may include a central processing unit 9100 and a memory 9140; the memory 9140 is coupled to the central processing unit 9100. It is worth noting that... Figure 3 This is an example; other types of structures can also be used to supplement or replace this structure to achieve telecommunications functions or other functions.

[0150] In one embodiment, the field number recognition method function can be integrated into the central processing unit 9100. The central processing unit 9100 can be configured to perform the following control:

[0151] Step S101: Receive video data and auxiliary recognition data, obtain multiple images to be recognized based on the video data, preprocess each image to be recognized to obtain a preprocessed image to be recognized, wherein the image to be recognized includes a digital sign to be recognized, the auxiliary recognition data includes thermal imaging information and depth data information, and the preprocessing includes channel conversion processing, illumination adaptive enhancement processing and stain detection processing.

[0152] Step S102: Process the pre-processed image to be identified through the field number detection model to obtain the position information of the digital card to be identified, the initial content information and the confidence level corresponding to the initial content information. The initial content information with a confidence level exceeding the preset confidence threshold is determined as the number recognition result corresponding to the unit digit of the digital card to be identified. The numbers in the digital card to be identified are used to represent the field number.

[0153] Step S103: Determine the temporal information corresponding to the pre-processed images to be identified in multiple adjacent frames. Based on the position information and temporal information of the digital card, determine the identification results of the number other than the last digit of the digital card to be identified. Combine the identification results of the number corresponding to the last digit and the identification results of the number other than the last digit to obtain the initial column number identification result corresponding to the digital card to be identified. Verify the initial column number identification result based on the auxiliary identification data. Determine the successfully verified initial column number identification result as the column number identification result.

[0154] As described above, the electronic device provided in this application embodiment innovatively receives video data and auxiliary recognition data, obtains multiple images to be recognized based on the video data, preprocesses each image to be recognized to obtain a preprocessed image to be recognized, wherein the image to be recognized includes a digital sign to be recognized, the auxiliary recognition data includes thermal imaging information and depth data information, the preprocessing includes channel conversion processing, illumination adaptive enhancement processing and stain detection processing, processes the preprocessed image to be recognized through a column number detection model to obtain the digital sign position information, initial content information and confidence level corresponding to the initial content information, determines the initial content information with a confidence level exceeding a preset confidence threshold as the number recognition result corresponding to the last digit of the digital sign to be recognized, and determines the number recognition result other than the last digit of the digital sign to be recognized based on the temporal information corresponding to the preprocessed images of adjacent frames, the digital sign position information and the temporal information, thereby obtaining the number recognition result corresponding to the digital sign to be recognized. This method effectively addresses the shortcomings of uneven lighting, dirt occlusion, changes in the style of number signs, and environmental interference in the identification of column numbers, resulting in low recognition accuracy and poor robustness, and significantly improves the accuracy and robustness of column number identification.

[0155] In another embodiment, the field number recognition device can be configured separately from the central processing unit 9100. For example, the field number recognition device can be configured as a chip connected to the central processing unit 9100, and the field number recognition method function can be implemented through the control of the central processing unit.

[0156] like Figure 3 As shown, the electronic device 9600 may further include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is worth noting that the electronic device 9600 does not necessarily need to include these components. Figure 3 All components shown; in addition, the electronic device 9600 may also include Figure 3 For components not shown, please refer to existing technologies.

[0157] like Figure 3 As shown, the central processing unit 9100, sometimes also referred to as a controller or operating control, may include a microprocessor or other processor device and / or logic device, which receives inputs and controls the operation of various components of the electronic device 9600.

[0158] The memory 9140 may be, for example, one or more of a cache, flash memory, hard drive, removable media, volatile memory, non-volatile memory, or other suitable devices. It may store the aforementioned failure-related information, and also store a program for executing that information. The central processing unit 9100 may execute the program stored in the memory 9140 to perform information storage or processing, etc.

[0159] Input unit 9120 provides input to central processing unit 9100. Input unit 9120 may be, for example, a keypad or touch input device. Power supply 9170 provides power to electronic device 9600. Display 9160 displays images and text. Display may be, for example, an LCD display, but is not limited thereto.

[0160] The memory 9140 can be a solid-state memory, such as a read-only memory (ROM), random access memory (RAM), a SIM card, etc. It can also be a memory that retains information even when power is off, can be selectively erased, and contains more data; examples of this type of memory are sometimes referred to as EPROMs. The memory 9140 can also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application / function storage unit 9142 for storing application programs and function programs or processes for executing the operation of the electronic device 9600 via the central processing unit 9100.

[0161] The memory 9140 may also include a data storage unit 9143 for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 9144 of the memory 9140 may include various drivers for the electronic device's communication functions and / or for performing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0162] The communication module 9110 is a transmitter / receiver that sends and receives signals via the antenna 9111. The communication module 9110 (transmitter / receiver) is coupled to the central processing unit 9100 to provide input signals and receive output signals, which is the same as in a conventional mobile communication terminal.

[0163] Based on different communication technologies, multiple communication modules 9110 can be configured in the same electronic device, such as cellular network modules, Bluetooth modules, and / or wireless LAN modules. The communication module 9110 (transmitter / receiver) is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide audio output via the speaker 9131 and receive audio input from the microphone 9132, thereby realizing typical telecommunications functions. The audio processor 9130 may include any suitable buffer, decoder, amplifier, etc. Additionally, the audio processor 9130 is coupled to a central processing unit 9100, enabling on-device recording via the microphone 9132 and on-device playback of stored audio via the speaker 9131.

[0164] Embodiments of this application also provide a computer-readable storage medium capable of implementing all steps of the field number identification method with the execution subject being a server or client in the above embodiments. The computer-readable storage medium stores a computer program that, when executed by a processor, implements all steps of the field number identification method with the execution subject being a server or client in the above embodiments. For example, when the processor executes the computer program, it implements the following steps:

[0165] Step S101: Receive video data and auxiliary recognition data, obtain multiple images to be recognized based on the video data, preprocess each image to be recognized to obtain a preprocessed image to be recognized, wherein the image to be recognized includes a digital sign to be recognized, the auxiliary recognition data includes thermal imaging information and depth data information, and the preprocessing includes channel conversion processing, illumination adaptive enhancement processing and stain detection processing.

[0166] Step S102: Process the pre-processed image to be identified through the field number detection model to obtain the position information of the digital card to be identified, the initial content information and the confidence level corresponding to the initial content information. The initial content information with a confidence level exceeding the preset confidence threshold is determined as the number recognition result corresponding to the unit digit of the digital card to be identified. The numbers in the digital card to be identified are used to represent the field number.

[0167] Step S103: Determine the temporal information corresponding to the pre-processed images to be identified in multiple adjacent frames. Based on the position information and temporal information of the digital card, determine the identification results of the number other than the last digit of the digital card to be identified. Combine the identification results of the number corresponding to the last digit and the identification results of the number other than the last digit to obtain the initial column number identification result corresponding to the digital card to be identified. Verify the initial column number identification result based on the auxiliary identification data. Determine the successfully verified initial column number identification result as the column number identification result.

[0168] As described above, the computer-readable storage medium provided in this application embodiment innovatively receives video data and auxiliary recognition data, obtains multiple images to be recognized based on the video data, preprocesses each image to be recognized to obtain a preprocessed image to be recognized, wherein the image to be recognized includes a digital sign to be recognized, the auxiliary recognition data includes thermal imaging information and depth data information, the preprocessing includes channel conversion processing, illumination adaptive enhancement processing and stain detection processing, processes the preprocessed image to be recognized through a column number detection model to obtain the digital sign position information, initial content information and confidence level corresponding to the initial content information, determines the initial content information with a confidence level exceeding a preset confidence threshold as the number recognition result corresponding to the last digit of the digital sign to be recognized, and determines the number recognition result other than the last digit of the digital sign to be recognized based on the temporal information corresponding to the preprocessed images to be recognized in adjacent frames, based on the digital sign position information and temporal information, thereby obtaining the number recognition result corresponding to the digital sign to be recognized. This method effectively addresses the shortcomings of uneven lighting, dirt occlusion, changes in the style of number signs, and environmental interference in the identification of column numbers, resulting in low recognition accuracy and poor robustness, and significantly improves the accuracy and robustness of column number identification.

[0169] Embodiments of this application also provide a computer program product capable of implementing all steps of the field number identification method in the above embodiments, where the execution subject is a server or a client. When this computer program / instruction is executed by a processor, it implements the steps of the field number identification method. For example, the computer program / instruction implements the following steps:

[0170] Step S101: Receive video data and auxiliary recognition data, obtain multiple images to be recognized based on the video data, preprocess each image to be recognized to obtain a preprocessed image to be recognized, wherein the image to be recognized includes a digital sign to be recognized, the auxiliary recognition data includes thermal imaging information and depth data information, and the preprocessing includes channel conversion processing, illumination adaptive enhancement processing and stain detection processing.

[0171] Step S102: Process the pre-processed image to be identified through the field number detection model to obtain the position information of the digital card to be identified, the initial content information and the confidence level corresponding to the initial content information. The initial content information with a confidence level exceeding the preset confidence threshold is determined as the number recognition result corresponding to the unit digit of the digital card to be identified. The numbers in the digital card to be identified are used to represent the field number.

[0172] Step S103: Determine the temporal information corresponding to the pre-processed images to be identified in multiple adjacent frames. Based on the position information and temporal information of the digital card, determine the identification results of the number other than the last digit of the digital card to be identified. Combine the identification results of the number corresponding to the last digit and the identification results of the number other than the last digit to obtain the initial column number identification result corresponding to the digital card to be identified. Verify the initial column number identification result based on the auxiliary identification data. Determine the successfully verified initial column number identification result as the column number identification result.

[0173] As described above, the computer program product provided in this application innovatively receives video data and auxiliary recognition data, obtains multiple images to be recognized based on the video data, preprocesses each image to be recognized to obtain a preprocessed image to be recognized, wherein the image to be recognized includes a digital sign to be recognized, the auxiliary recognition data includes thermal imaging information and depth data information, the preprocessing includes channel conversion processing, illumination adaptive enhancement processing and stain detection processing, processes the preprocessed image to be recognized through a column number detection model to obtain the digital sign position information, initial content information and confidence level corresponding to the initial content information, determines the initial content information with a confidence level exceeding a preset confidence threshold as the number recognition result corresponding to the last digit of the digital sign to be recognized, and determines the number recognition result other than the last digit of the digital sign to be recognized based on the temporal information corresponding to the preprocessed images of adjacent frames, and based on the digital sign position information and temporal information, thereby obtaining the number recognition result corresponding to the digital sign to be recognized. This method effectively addresses the shortcomings of uneven lighting, dirt occlusion, changes in the style of number signs, and environmental interference in the identification of column numbers, resulting in low recognition accuracy and poor robustness, and significantly improves the accuracy and robustness of column number identification.

[0174] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0175] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0176] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0177] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0178] Specific embodiments have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A method for identifying field numbers, characterized in that, The method includes: The system receives video data and auxiliary recognition data, obtains multiple images to be recognized based on the video data, preprocesses each image to be recognized, and obtains a preprocessed image to be recognized. The image to be recognized includes a digital sign to be recognized, the auxiliary recognition data includes thermal imaging information and depth data information, and the preprocessing includes channel conversion processing, illumination adaptive enhancement processing, and stain detection processing. The pre-processed image to be identified is processed by a field number detection model to obtain the position information, initial content information, and confidence level of the number card to be identified. The initial content information with a confidence level exceeding a preset confidence threshold is determined as the number recognition result corresponding to the last digit of the number card to be identified. The number in the number card to be identified is used to represent the field number. The process includes: when the difference between the number recognition results corresponding to the last digit of adjacent number cards to be identified is greater than 1, determining the motion trajectory of the number card to be identified in consecutive adjacent pre-processed images to be identified based on the position information of the number card; and based on the motion trajectory, completing the missing number recognition results by linear interpolation. The process involves determining the temporal information corresponding to multiple pre-processed frames of the image to be identified, identifying the number recognition results of the number card other than the last digit based on the position information and the temporal information, combining the number recognition results corresponding to the last digit with the number recognition results other than the last digit to obtain the initial column number recognition result corresponding to the number card to be identified, and verifying the initial column number recognition result based on the auxiliary recognition data. This includes: determining the material characteristics of the number card one by one through the thermal imaging information, identifying the number card whose material characteristics are different from the preset material characteristics as false detection number cards; determining the spatial arrangement order of the number cards to be identified based on the depth data information, comparing the continuity between the spatial arrangement order and the initial column number recognition result to obtain a comparison result; determining the current initial column number recognition result as successfully verified when the comparison result is successful and there are no false detection number cards; and determining the successfully verified initial column number recognition result as the column number recognition result.

2. The method according to claim 1, characterized in that, The step of preprocessing each of the images to be identified to obtain preprocessed images to be identified includes: Based on the thermal imaging information and the preset illumination level, the illumination level corresponding to each of the images to be identified is determined. The preset illumination level includes low illumination level, normal illumination level and high illumination level. The region corresponding to the low illumination level in each of the images to be identified is enhanced by histogram equalization, and the region corresponding to the high illumination level in each of the images to be identified is processed by dynamic range compression to obtain the image to be identified with processed illumination. If a shadow is detected in the area where the digital sign to be identified is located in the image to be identified after processing the lighting, the initial content information in the digital sign to be identified is completed by a generative adversarial network.

3. The method according to claim 1, characterized in that, The step of preprocessing each of the images to be identified to obtain preprocessed images to be identified includes: The outline information of the digital sign to be identified is extracted by edge detection. If the outline information does not meet the preset connectivity conditions, the digital sign to be identified is identified as the digital sign to be identified that is covered by stains. The preset connectivity conditions include the outline line obtained based on the outline information being broken and / or the outline line obtained based on the current outline information being different from the preset outline line. The digit sign to be identified that is obscured by stains is processed by context repair method to obtain the repaired digit sign.

4. The method according to claim 1, characterized in that, Also includes: When the pattern of the digital card to be identified or the environment in which the digital card to be identified is changed, the digital card to be identified that is not successfully identified is identified as a difficult sample. The model parameters corresponding to the field number detection model are updated based on the difficult samples to obtain the updated field number detection model.

5. The method according to claim 1, characterized in that, After obtaining the comparison results, the following is also included: If the comparison result is a failure and / or there is a false detection of the digital card, the current initial field number identification result is determined to be a failed verification. Remove the image to be identified corresponding to the falsely detected number sign, and repeat the steps of processing the pre-processed image to be identified through the column number detection model to obtain the number sign position information, initial content information and the confidence level corresponding to the initial content information of the number sign to be identified.

6. A field number identification device, characterized in that, The device includes: The receiving module is used to receive video data and auxiliary recognition data, obtain multiple images to be recognized based on the video data, preprocess each image to be recognized to obtain a preprocessed image to be recognized, wherein the image to be recognized includes a digital sign to be recognized, the auxiliary recognition data includes thermal imaging information and depth data information, and the preprocessing includes channel conversion processing, illumination adaptive enhancement processing and stain detection processing. The recognition module is used to process the pre-processed image to be recognized through a column number detection model to obtain the position information of the number card to be recognized, the initial content information, and the confidence level corresponding to the initial content information. The initial content information whose confidence level exceeds a preset confidence threshold is determined as the number recognition result corresponding to the last digit of the number card to be recognized. The number in the number card to be recognized is used to represent the column number. The module includes: when the difference between the number recognition results corresponding to the last digit of adjacent number cards to be recognized is greater than 1, determining the motion trajectory of the number card to be recognized in consecutively adjacent pre-processed images based on the number card position information; and filling in the missing number recognition results based on the motion trajectory using linear interpolation. The output module is used to determine the temporal information corresponding to the pre-processed images to be identified in multiple adjacent frames, and based on the position information of the digital sign and the temporal information, to determine the identification result of the number other than the last digit of the digital sign to be identified, and to combine the identification result of the number corresponding to the last digit and the identification result of the number other than the last digit to obtain the initial column number identification result corresponding to the digital sign to be identified, and to verify the initial column number identification result based on the auxiliary identification data, including: determining the material characteristics of the digital sign to be identified one by one through the thermal imaging information, and identifying the digital sign to be identified whose material characteristics are different from the preset material characteristics as false detection digital signs; determining the spatial arrangement order of the digital signs to be identified based on the depth data information, comparing the continuity of the spatial arrangement order and the initial column number identification result to obtain a comparison result; when the comparison result is a successful comparison and there are no false detection digital signs, determining the current initial column number identification result as a successful verification; and determining the successfully verified initial column number identification result as the column number identification result.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the field number recognition method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the field number identification method according to any one of claims 1 to 5.

Images