Deep learning-based picture frame information intelligent identification method and system, and storage medium

Abstract

The invention discloses a deep learning-based picture frame information intelligent identification method and system and a storage medium, and the method comprises the steps: extracting a picture frame range under a layout, and filtering the picture frame range to obtain a target picture frame range; on the basis of a graph recognition algorithm, identifying a picture tag bar cell in each target picture frame range, and predicting a whole picture tag bar area on the basis of a yov8 prediction model; and on the basis of the picture tag bar cells and the text bounding box, determining picture tag information in the target picture frame range. According to the method, graph and text information in the DWG file is automatically identified and analyzed through advanced graph and image processing and deep learning technologies, key data can be rapidly and accurately extracted from a large number of drawings, and the efficiency and quality of building design and construction are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of drawing recognition, and in particular to a method, system and storage medium for intelligent recognition of drawing frame information based on deep learning. Background Art

[0002] In the construction industry, the design and construction process is highly dependent on accurate and detailed drawings. DWG format, as a widely used CAD file format, contains important information of construction projects, such as floor plans, elevations, and sections. However, as the complexity of construction projects increases, manually managing and analyzing these drawings becomes increasingly time-consuming and laborious, especially when specific data needs to be extracted from a large number of drawings or error checks need to be performed. Traditional methods rely on visual inspection and manual input by professionals, which is not only inefficient but also prone to errors. In addition, as the scale of construction projects increases and modifications are frequent, maintaining the accuracy and up-to-date status of drawings becomes a challenge. These factors have prompted the need for smarter, automated solutions to improve the efficiency and accuracy of processing drawings. Summary of the invention

[0003] The technical problem to be solved by the present invention is that, in view of the above-mentioned defects of the prior art, a method, system and storage medium for intelligent recognition of drawing frame information based on deep learning are provided, aiming to solve the problem in the prior art that as the complexity of construction projects increases, manual management and analysis of drawings become more and more time-consuming and labor-intensive, especially when it is necessary to extract specific data from a large number of drawings or perform error checks, it relies on visual inspection and manual input by professionals, which is inefficient and difficult to ensure accuracy.

[0004] In order to solve the above technical problems, the technical solution adopted by the present invention is as follows:

[0005] In a first aspect, the present invention provides a method for intelligently identifying frame information based on deep learning, wherein the method comprises:

[0006] Extracting a frame range under the layout, and filtering the frame range to obtain a target frame range;

[0007] Identify each label title bar cell in the target frame range based on a graphic recognition algorithm, and predict the entire label bar area based on a yolv8 prediction model;

[0008] Based on the label title bar cells and the text bounding box, the label information in the target frame range is determined.

[0009] In one implementation, filtering the frame range to obtain a target frame range includes:

[0010] Preliminarily filtering the closed path data within the frame range;

[0011] Acquire text features and graphic features in the frame range that have been initially filtered, wherein the text features reflect the text objects in the frame range, and the graphic features reflect the tables and inner frames in the frame range;

[0012] Based on the text features and the graphic features, a frame range that meets the conditions is determined to obtain the target frame range.

[0013] In one implementation, the training method of the yolov8 prediction model includes:

[0014] Obtain several frame images, and divide the frame images into a training set, a validation set, and a test set, wherein the training set and the validation set are annotated with labelimg data to obtain image label categories, and each frame image corresponds to at most one image label category;

[0015] The yolov8 model is trained based on the training set, and hyperparameters are set. Based on the hyperparameters, the yolov8 model is controlled to reduce the focus on classification and increase the focus on the label bounding box during the training process. After the training is completed, the yolov8 prediction model is obtained.

[0016] In one implementation, the training method of the yolov8 prediction model further includes:

[0017] Performing data enhancement on the training set, wherein the data enhancement method includes offline enhancement and online enhancement, wherein the offline enhancement refers to completing all data enhancement operations on a computer at one time before model training, including: converting the original training set into an extended data set containing multiple enhanced versions, saving the extended data set to a disk or memory, and using the extended data set for model training;

[0018] The online enhancement means that the model will perform data enhancement operations in real time during the model training process, including: in each iteration, after extracting a sample from the original training set, it is immediately subjected to a series of random transformations to generate a new enhanced sample, which is then used to train the current batch.

[0019] In one implementation, determining the label information in the target frame range based on the label title bar cell and the text bounding box includes:

[0020] Determine the confidence information corresponding to the text bounding box, the confidence information corresponds to the cell of the label title bar, and the text bounding box reflects the text area in the label bar area;

[0021] Evaluate the cells of the title bar of the graphic label, and select at most three cells of the title bar of the graphic label whose confidence information exceeds a threshold;

[0022] Based on the filtered label title bar cells, the label information in the target frame range is determined.

[0023] In one implementation, determining the label information in the target frame range based on the label title bar cell and the text bounding box includes:

[0024] According to the principle of prioritizing cells and then adjacent cells, the text in the label title cells is filtered and merged;

[0025] Based on the Qwen_VL model's understanding of the text, the label value of each label title cell is obtained;

[0026] Based on the matching degree between the image tag value and the text in the image tag title cell, a final image tag value is determined, and the image tag title cell corresponding to the final image tag value is determined to obtain the image tag information.

[0027] In one implementation, determining the label information in the target frame range based on the label title bar cell and the text bounding box includes:

[0028] If there is no label title bar cell whose confidence information exceeds the threshold, the Qwen_VL multimodal model is used to predict the text in the label title bar cell.

[0029] In a second aspect, an embodiment of the present invention provides a frame information intelligent recognition system based on deep learning, wherein the system comprises:

[0030] A frame range analysis module is used to extract the frame range under the layout and filter the frame range to obtain a target frame range;

[0031] A graphic image analysis module, used to identify the label title bar cells in each target frame range based on a graphic recognition algorithm, and to predict the entire label bar area based on a yolov8 prediction model;

[0032] The label information analysis module is used to determine the label information in the target frame range based on the label title bar cells and the text bounding box.

[0033] In a third aspect, an embodiment of the present invention further provides a terminal, wherein the terminal includes a memory, a processor, and a deep learning-based intelligent frame information recognition program stored in the memory and executable on the processor, and when the processor executes the deep learning-based intelligent frame information recognition program, the steps of the deep learning-based intelligent frame information recognition method of any one of the above-mentioned schemes are implemented.

[0034] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein a deep learning-based intelligent frame information recognition program is stored on the computer-readable storage medium, and when the deep learning-based intelligent frame information recognition program is executed by a processor, the steps of the deep learning-based intelligent frame information recognition method described in any one of the above-mentioned schemes are implemented.

[0035] Beneficial effect: Compared with the prior art, the present invention provides a method for intelligently identifying frame information based on deep learning. The present invention first extracts the frame range under the layout, and filters the frame range to obtain the target frame range, thereby avoiding unnecessary calculation of non-frame areas, thereby improving the overall efficiency, reducing the false detection rate by filtering, and ensuring that the target frame range finally processed is highly relevant; then, based on the graphic recognition algorithm, identify each label title bar cell in the target frame range, predict the entire label bar area based on the yolov8 prediction model, and improve the text confidence of the bounding box in the area; and use this prediction box to cut the frame png, so that the Qwen_VL multimodal model can understand it, thereby reducing the multimodal model reasoning resource consumption, eliminating interference from other irrelevant areas of the frame, and providing important clues for understanding the context of the frame; then, based on the label title bar cell and the text bounding box, determine the label information in the target frame range. In this way, key data can be quickly and accurately extracted from a large number of drawings, thereby improving the efficiency and quality of architectural design and construction. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A flowchart of a specific implementation of a method for intelligently identifying frame information based on deep learning provided in an embodiment of the present invention.

[0037] Figure 2 A flow chart of obtaining a target frame range in a method for intelligently identifying frame information based on deep learning provided in an embodiment of the present invention.

[0038] Figure 3 A flowchart for determining label information in a target frame range in a method for intelligently identifying frame information based on deep learning provided in an embodiment of the present invention.

[0039] Figure 4 An overall flow chart of image signature recognition in the deep learning-based intelligent recognition method for image frame information provided in an embodiment of the present invention.

[0040] Figure 5 A principle block diagram of a deep learning-based intelligent recognition system for frame information provided in an embodiment of the present invention.

[0041] Figure 6 A functional block diagram of a terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0043] As construction projects increase in complexity, manually managing and analyzing these drawings becomes increasingly time-consuming and laborious, especially when specific data needs to be extracted from a large number of drawings or error checking is required. Traditional methods rely on visual inspection and manual input by professionals, which is not only inefficient but also prone to errors. In addition, as construction projects grow in size and revisions are frequent, maintaining the accuracy and up-to-dateness of drawings becomes a challenge.

[0044] In order to solve the above problems, the present embodiment provides a method for intelligent recognition of frame information based on deep learning. In specific implementation, the present embodiment first extracts the frame range under the layout, and filters the frame range to obtain the target frame range, thereby avoiding unnecessary calculation of non-frame areas, thereby improving the overall efficiency, reducing the false detection rate by filtering, and ensuring that the target frame range finally processed is highly relevant; then, based on the graphic recognition algorithm, identify each label title bar cell in the target frame range, predict the entire label bar area based on the yolov8 prediction model, and improve the text confidence of the bounding box in the area; and use this prediction box to cut the frame png, so that the Qwen_VL multimodal model can understand it, thereby reducing the multimodal model reasoning resource consumption and eliminating interference from other irrelevant areas of the frame; then, based on the label title bar cell and the text bounding box, determine the label information in the target frame range. In this way, key data can be quickly and accurately extracted from a large number of drawings, thereby improving the efficiency and quality of architectural design and construction.

[0045] For example, in practical applications, suppose there is a construction project of a large commercial complex, which contains hundreds of design drawings, each of which has multiple frames, and each frame contains detailed architectural elements and annotations. Using the traditional manual management method, engineers need to check these frames one by one and record the label information, which is an extremely time-consuming and error-prone process. At this time, the intelligent recognition method of frame information based on deep learning of this embodiment can be adopted. First, the frame range under the layout is extracted, and the frame range is filtered to obtain the target frame range; then, the label title bar cell in each target frame range is identified based on the graphic recognition algorithm, and the entire label bar area is predicted based on the yolov8 prediction model; finally, based on the label title bar cell and the text bounding box, the label information in the target frame range is determined, which is convenient for architects and engineers to consult and analyze, and also convenient for project managers to track progress and control costs. This method simplifies the drawing management process, improves work efficiency, reduces human errors, ensures the accuracy and timely update of the information of the construction project, and provides strong support for the architectural design and construction phase.

[0046] The present embodiment provides a method for intelligently identifying frame information based on deep learning, which can be applied to smart terminals, such as Figure 1 As shown in , the specific steps include:

[0047] Step S100: extract the frame range under the layout, and filter the frame range to obtain the target frame range.

[0048] In this embodiment, a hash table is first maintained for each layout of the drawing, and the hash table uses the XOR hash value of the layer name and the block name as the key to accurately classify the line primitives. This strategy makes full use of the characteristics that the outer boundary lines of the frame are usually in the same block and belong to the same layer, effectively filters out irrelevant interference lines, and ensures the accurate extraction of the outer boundary path of the frame; then, the CGALArrangement algorithm is used, with the help of its efficient half-edge data structure, to facilitate the access to edges, faces, and vertices, and this algorithm removes "bridges" (non-closed paths) and "antennas" (discontinuous paths) during encapsulation, and only retains those bounded and closed paths, while further clearing the internal loops in the circular path, so as to ensure that the final screened frame is both complete and non-redundant; finally, in order to further improve processing efficiency and accuracy, the box_intersection_d algorithm of CGAL is used, which can significantly optimize the collision detection process and ensure that the extraction of the entire frame range is both fast and accurate. Through this series of steps, this embodiment can efficiently and accurately extract the frame range under the layout, while eliminating any possible interference factors, ensuring the quality and reliability of the final result. After extracting the frame range under the layout, the frame range is filtered to finally obtain the target frame range, ensuring that the object of subsequent analysis and processing is the area that truly contains data or chart information, improving the overall recognition efficiency and accuracy, and avoiding unnecessary waste of computing resources.

[0049] In one implementation, Figure 2 As shown in , step S100 specifically includes the following steps:

[0050] Step S101, preliminarily filtering the closed path data in the frame range;

[0051] Step S102, obtaining text features and graphic features in the frame range that has been initially filtered, wherein the text features reflect the text objects in the frame range, and the graphic features reflect the tables and inner frames in the frame range;

[0052] Step S103: determining a frame range that meets the conditions based on the text features and the graphic features, and obtaining the target frame range.

[0053] In one implementation, after extracting the frame range under the layout, this embodiment first performs a preliminary filtering of the closed path data in the frame range. This stage mainly focuses on the geometric features of the frame, such as whether the frame is orthogonal, that is, whether the edges are vertical or parallel; whether the aspect ratio falls within a preset range to exclude frames that are too flat or slender; and evaluating the number of line primitives inside the frame, which helps to identify frames that are more likely to be target objects, thereby greatly improving the efficiency of subsequent processing through simple feature filtering; then, the text features are analyzed, and the text features reflect the text objects in the frame range. Optical character recognition (OCR) technology is used to analyze whether the frame contains specific keywords or phrases, such as "engineering", "countersignature" or "Project". By recording the number of occurrences of these fixed expressions, the search scope is further narrowed to focus on those that are more likely to be related to the industry. Then, the graphic features are evaluated, the graphic features reflect the table and inner frame in the frame range, and the graphic recognition algorithm is used to detect whether there is a table structure or an embedded sub-frame in the frame. The graphic recognition algorithm includes but is not limited to OpenCV, convolutional neural network, deep neural network and random forest, which helps to distinguish complex frame types for subsequent more accurate data processing; finally, the obtained text features and graphic features are combined, and the relationship between different frames is determined by performing Boolean operations, such as determining whether two or more frames completely contain each other, or whether there is an intersection between them. On this basis, according to the needs of business logic, it is decided which frames should be retained and which can be eliminated, and finally the target frame range that meets the conditions is determined, which can ensure that the final frame set meets both the screening criteria and specific business needs.

[0054] Step S200, identifying each label title bar cell in the target frame range based on a graphic recognition algorithm, and predicting the entire label bar area based on a yolov8 prediction model.

[0055] In this embodiment, after obtaining the target frame range, first, identify each label title bar cell in the target frame range based on the graphic recognition algorithm. Then, predict the entire label bar area based on the yolov8 prediction model: First, obtain 3380 frame images, and divide the collected frame images into training set, verification set and test set in a ratio of 8:1:1. The training set has 2704 images, including 500 background images (no labeled instances), and the verification set has 338 images, including 80 background images and 338 test sets. The training set and the verification set are accurately labeled by labelimg software and classified as ["label"] to ensure that each frame image is associated with only one label category to build a clear classification standard. Then, according to the frame range, use OpenGL to display the image, and only retain the part of interest within the frame range. Next, to prevent overfitting, a small model yolov8n is selected for training from scratch, and hyperparameters are set. Because there is only one category, the yolov8 model is controlled based on the hyperparameters to reduce the focus on classification during training and increase the focus on the bounding box of the label. In addition, data enhancement is a very critical step in the training process. The data enhancement method described in this embodiment includes offline enhancement and online enhancement, wherein the offline enhancement refers to completing all data enhancement operations on a computer at one time before model training, including: converting the original training set into an expanded data set containing multiple enhanced versions, and saving the expanded data set to a disk or memory, and using the expanded data set for model training; the online enhancement refers to the model performing data enhancement operations in real time during the model training process, including: each time a sample is extracted from the original training set, a series of random transformations are immediately performed on it to generate a new enhanced sample, and then the enhanced sample is used to train the current batch. After the above training is completed, the yolov8 prediction model is obtained, and the entire label column area is finally predicted based on the obtained yolov8 prediction model. In summary, this embodiment uses a graphic recognition algorithm to accurately locate and identify the cells in the label title bar, which helps to quickly lock the key descriptive information of the frame, while the yolov8 prediction model, with its excellent target detection capability, predicts the entire label bar area, improves the confidence of the text in the area, and serves as a data source for multimodal understanding. This two-pronged strategy improves the accuracy and efficiency of information recognition in the frame, and also provides a solid foundation for the subsequent comprehensive analysis of label information.

[0056] Step S300: Determine the label information in the target frame range based on the label title bar cells and the text bounding box.

[0057] In this embodiment, after obtaining the cell of the title bar of the image label and the text bounding box, the image label information in the target frame range is determined based on the obtained cell of the title bar of the image label and the text bounding box.

[0058] This embodiment achieves accurate positioning and identification of the label information in the target frame by integrating the information of the label title bar cells and text bounding boxes obtained in the previous steps, where the text bounding box reflects the text position in the label bar area. This process not only improves the accuracy of label content parsing, but also ensures that the semantic information of each label can be effectively separated and understood even in complex layouts, thereby improving the efficiency and reliability of application scenarios such as automated document analysis, engineering drawing interpretation, or map annotation. This comprehensive analysis method overcomes the limitations of traditional image processing technology when faced with diverse and unstructured data, and provides a solid foundation for subsequent data understanding and intelligent decision-making.

[0059] In one implementation, Figure 3 As shown, the step S300 specifically includes:

[0060] Step S301, determining the confidence information corresponding to the text bounding box, wherein the confidence information corresponds to the cell of the title bar of the graphic label;

[0061] Step S302, evaluating the cells of the title bar of the graphic label, and selecting at most three cells of the title bar of the graphic label whose confidence information exceeds a threshold;

[0062] Step S303: Determine the label information in the target frame range based on the filtered label title bar cells.

[0063] In one implementation, the overall flow chart of the image signature recognition in this embodiment is as follows: Figure 4As shown. First, this embodiment determines the confidence information corresponding to the text bounding box through several important dimensions, including the maximum continuous common substring, substring validity, cell and text object bounding box intersection ratio, unmatched characters and yolov8 training model prediction confidence. The continuous common substring refers to the common substring of the text value string in the drawing and the label title. When establishing a control group, it will be established in the form of a set. For example, the expression of the project name includes but is not limited to: project name ProjectTitle, project name Proj, each control will participate in the confidence evaluation, and the one with the highest confidence will be taken as the title of the label entry; the confidence contribution of this dimension is Confidence + = len*len*2.0, where len is the maximum continuous common substring length. The substring validity is to compare the common substring with the original text. For example, in a large text, if a small part of the text string constitutes a common substring, its confidence will be reduced; the confidence contribution of this dimension is Confidence + = (len / srcLen) * (len / srcLen) * 4.0, where srcLen is the length of the original text string. The confidence contribution of the intersection-union ratio dimension of the cell and the text object bounding box is Confidence+=I / U*1.5, where I is the intersection area and U is the union area; the unmatched characters, such as the project name entry, the optimal control text is: project name Project, if two text objects {project, Project} are queried in a cell, the unmatched character is "name", and rewards or penalties are given according to the unmatched rate; the confidence contribution is O=noMatchLen / tagFactorLen; O>=0.4999:Confidence-=10*noMatchLen; 0.2499<=O<0.4999:Confidence-=5*noMatchLen; 0.1999<=O<0.2499:Confidence-=2*noMatchLen; O<0.1999:Confidence+=3*len, where noMatchLen is the number of unmatched characters and tagFactorLen is the length of the control text string. The yolov8 training model predicts confidence. When the text bounding box is within the prediction box, the confidence contribution is: Confidence+=conf*10, where conf is the prediction box confidence. The confidence information corresponds to the label title bar cell. For example, there are five possible label title bar cells, namely A, B, C, D, and E, where A has a confidence of 95%, B has a confidence of 88%, C has a confidence of 75%, D has a confidence of 60%, and E has a confidence of 45%. The confidence information is used to determine which text bounding boxes and related label title bar cells are reliable, thereby improving the accuracy and reliability of the entire system in image text recognition tasks.

[0064] Next, set a confidence threshold, such as 80%, to evaluate and screen the label title bar cells, and screen out at most three label title bar cells whose confidence information exceeds the threshold, such as the label title bar cells A and B mentioned above; if there are no label title bar cells whose confidence information exceeds the threshold, use the Qwen_VL multimodal model to predict the text in the label title bar cells. Then, if there are label title bar cells whose confidence information exceeds the threshold, that is, after screening out at most three label title bar cells whose confidence information exceeds the threshold, the text in the label title cells is screened and merged according to the principle of giving priority to cells and then adjacent cells. For example, if there is " in label title bar cell A, Figure 1 :Pier Design”, while the adjacent cell of the title bar of another drawing label has “ Figure 2 :bridge deck structure", even if the latter has a slightly lower confidence, it will be merged into the information in cell A of the label title bar. At this time, the text in cell A of the label title bar is " Figure 1 :Bridge pier design, Figure 2 :bridge deck structure”, because they are closely related in semantics. This strategy can better maintain the integrity of information. Then, based on the Qwen_VL model’s understanding of the text, the label value of each label title cell is obtained. For example, based on the Qwen_VL model’s understanding of the text in the label title bar cell A, Figure 1 :Bridge pier design, Figure 2 :Bridge deck structure”, it is concluded that the tag value of the tag title bar cell A is “Bridge Design Overview”. Finally, based on the degree of match between the tag value and the text in the tag title cell, the final tag value is determined, and the tag title cell corresponding to the final tag value is determined to obtain the tag information. If the degree of match between the text in at most three tag title bar cells and the tag value is greater than the threshold, the highest one is taken as the final tag value; if the degree of match between the text in at most three tag title bar cells and the tag value is less than the threshold, the text in the tag title bar cell with the highest confidence is taken as the final tag value; if there is no tag title bar cell whose confidence information exceeds the threshold, the predicted value of the text in the tag title bar cell predicted by the Qwen_VL multimodal model is taken as the final tag value. For example, the matching threshold is set to 0.8, and the text in the tag title bar cell A is “ Figure 1 :Bridge pier design, Figure 2 The matching degree between the label value "Bridge Design Overview" in the label title bar cell A is 0.9, and the text " Figure 2 The match degree between ": Bridge abutment construction details" and the label value "Material list" in the label title bar cell B is 0.85, because the text in the label title bar cell A matches the label value in the label title bar cell A more closely, so " Figure 1 :Bridge pier design, Figure 2 :Bridge deck structure" becomes the final drawing label value. This process ensures the accuracy of the drawing label information, while also providing a structured understanding of the drawing content, greatly facilitating subsequent data processing and information retrieval.

[0065] In summary, this embodiment first extracts the frame range under the layout, and filters the frame range to obtain the target frame range, avoiding unnecessary calculations of non-frame areas, thereby improving overall efficiency, reducing the false detection rate through filtering, and ensuring that the target frame range finally processed is highly relevant; then, based on the graphic recognition algorithm, identify each label title bar cell in the target frame range, predict the entire label bar area based on the yolov8 prediction model, and improve the text confidence of the bounding box in the area; and use this prediction box to cut the frame png, so that the Qwen_VL multimodal model can understand it, thereby reducing the multimodal model reasoning resource consumption and eliminating interference from other irrelevant areas of the frame; then, based on the label title bar cell and the text bounding box, determine the label information in the target frame range. In this way, key data can be quickly and accurately extracted from a large number of drawings, thereby improving the efficiency and quality of architectural design and construction.

[0066] like Figure 5 As shown in, the present embodiment also provides a frame information intelligent recognition system based on deep learning, the system comprising: a frame range analysis module 10, a graphic image analysis module 20 and a label information analysis module 30. Specifically, the frame range analysis module 10 is used to extract the frame range under the layout, and filter the frame range to obtain the target frame range. The graphic image analysis module 20 is used to identify each label title bar cell in the target frame range based on a graphic recognition algorithm, and predict the entire label bar area based on the yolov8 prediction model. The label information analysis module 30 is used to determine the label information in the target frame range based on the label title bar cell and the text bounding box.

[0067] In one implementation, the frame range analysis module 10 includes:

[0068] A filtering unit, used for preliminarily filtering the closed path data in the frame range;

[0069] A text feature and graphic feature acquisition unit, used to acquire text features and graphic features in the preliminarily filtered frame range, wherein the text features reflect the text objects in the frame range, and the graphic features reflect the tables and inner frames in the frame range;

[0070] The target frame range acquisition unit determines a frame range that meets the conditions based on the text features and the graphic features, and obtains the target frame range.

[0071] In one implementation, the graphic image analysis module 20 includes:

[0072] The data set preparation unit obtains a number of frame images and divides the frame images into a training set, a validation set, and a test set, wherein the training set and the validation set are annotated with labelimg data to obtain image label categories, and each frame image corresponds to at most one image label category;

[0073] A hyperparameter setting unit, which trains the yolov8 model based on the training set and sets hyperparameters, and controls the yolov8 model to reduce the focus on classification and increase the focus on the image label boundary box during the training process based on the hyperparameters, and obtains the yolov8 prediction model after the training is completed;

[0074] A data enhancement unit is used to perform data enhancement on the training set. The data enhancement methods include offline enhancement and online enhancement. The offline enhancement refers to completing all data enhancement operations on a computer at one time before model training, including: converting the original training set into an extended data set containing multiple enhanced versions, saving the extended data set to a disk or memory, and using the extended data set for model training. The online enhancement refers to the real-time data enhancement operation of the model during the model training process, including: in each iteration, after extracting a sample from the original training set, immediately subjecting it to a series of random transformations to generate a new enhanced sample, and then using the enhanced sample to train the current batch.

[0075] In one implementation, the image label information analysis module 30 includes:

[0076] A confidence determination unit, used to determine confidence information corresponding to the text bounding box, wherein the confidence information corresponds to a cell of the title bar of the graphic label;

[0077] A screening unit, used for evaluating the cells of the title bar of the graphic label, and screening out at most three cells of the title bar of the graphic label whose confidence information exceeds a threshold;

[0078] A label information determining unit, which determines the label information in the target frame range based on the filtered label title bar cells;

[0079] The screening unit comprises:

[0080] The text prediction subunit is used to predict the text in the cell of the title bar of the image label using the Qwen_VL multimodal model if there is no cell of the title bar of the image label whose confidence information exceeds the threshold;

[0081] The text filtering and merging unit is used to filter and merge the text in the label title cell according to the principle of first cell and then adjacent cells;

[0082] The image label information determining unit includes:

[0083] A label value acquisition unit, used to obtain the label value of each label title cell based on the understanding of the text by the Qwen_VL model;

[0084] The tag information acquisition unit is used to determine the final tag value based on the matching degree between the tag value and the text in the tag title cell, and determine the tag title cell corresponding to the final tag value to obtain the tag information.

[0085] The working principles of each module in the deep learning-based frame information intelligent recognition system of this embodiment are the same as the principles of each step in the above method embodiment, and will not be repeated here.

[0086] Based on the above embodiment, the present invention further provides a terminal, the principle block diagram of the terminal can be as follows: Figure 6 The terminal may include one or more processors 100 ( Figure 6 Only one is shown in the figure), a memory 101 and a computer program 102 stored in the memory 101 and executable on one or more processors 100, for example, a program for intelligent recognition of frame information based on deep learning. When one or more processors 100 execute the computer program 102, each step in the embodiment of the method for intelligent recognition of frame information based on deep learning can be implemented. Alternatively, when one or more processors 100 execute the computer program 102, the functions of each module / unit in the embodiment of the method for intelligent recognition of frame information based on deep learning can be implemented, which is not limited here.

[0087] In one embodiment, the processor 100 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0088] In one embodiment, the memory 101 may be an internal storage unit of an electronic device, such as a hard disk or memory of the electronic device. The memory 101 may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device. Further, the memory 101 may also include both an internal storage unit of the electronic device and an external storage device. The memory 101 is used to store computer programs and other programs and data required by the terminal. The memory 101 may also be used to temporarily store data that has been output or is to be output.

[0089] Those skilled in the art will understand that Figure 6 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the terminal to which the scheme of the present invention is applied. The specific terminal may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0090] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, operating database or other media used in the embodiments provided by the present invention may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double operational data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0091] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for intelligent recognition of frame information based on deep learning, characterized in that: The method comprises: Extracting a frame range under the layout, and filtering the frame range to obtain a target frame range; Identify each label title bar cell in the target frame range based on a graphic recognition algorithm, and predict the entire label bar area based on a yolov8 prediction model; Based on the label title bar cells and the text bounding box, the label information in the target frame range is determined.

2. The method for intelligently identifying frame information based on deep learning according to claim 1, characterized in that: The filtering of the frame range to obtain a target frame range includes: Preliminarily filtering the closed path data within the frame range; Acquire text features and graphic features in the frame range that have been initially filtered, wherein the text features reflect the text objects in the frame range, and the graphic features reflect the tables and inner frames in the frame range; Based on the text features and the graphic features, a frame range that meets the conditions is determined to obtain the target frame range.

3. The method for intelligently identifying frame information based on deep learning according to claim 1, characterized in that: The training method of the yolov8 prediction model comprises: Obtain several frame images, and divide the frame images into a training set, a validation set, and a test set, wherein the training set and the validation set are annotated with labelimg data to obtain image label categories, and each frame image corresponds to at most one image label category; The yolov8 model is trained based on the training set, and hyperparameters are set. Based on the hyperparameters, the yolov8 model is controlled to reduce the focus on classification and increase the focus on the label bounding box during the training process. After the training is completed, the yolov8 prediction model is obtained.

4. The method for intelligently identifying frame information based on deep learning according to claim 3, characterized in that: The training method of the yolov8 prediction model also includes: Performing data enhancement on the training set, wherein the data enhancement method includes offline enhancement and online enhancement, wherein the offline enhancement refers to completing all data enhancement operations on a computer at one time before model training, including: converting the original training set into an extended data set containing multiple enhanced versions, saving the extended data set to a disk or memory, and using the extended data set for model training; The online enhancement means that the model will perform data enhancement operations in real time during the model training process, including: in each iteration, after extracting a sample from the original training set, it is immediately subjected to a series of random transformations to generate a new enhanced sample, which is then used to train the current batch.

5. The method for intelligently identifying frame information based on deep learning according to claim 1, characterized in that: The step of determining the label information in the target frame range based on the label title bar cell and the text bounding box includes: Determine the confidence information corresponding to the text bounding box, the confidence information corresponds to the cell of the label title bar, wherein the text bounding box reflects the text area in the label bar area; Evaluate the cells of the title bar of the graphic label, and select at most three cells of the title bar of the graphic label whose confidence information exceeds a threshold; Based on the filtered label title bar cells, the label information in the target frame range is determined.

6. The method for intelligently identifying frame information based on deep learning according to claim 5, characterized in that: The step of determining the label information in the target frame range based on the label title bar cell and the text bounding box includes: According to the principle of prioritizing cells and then adjacent cells, the text in the label title cells is filtered and merged; Based on the Qwen_VL model's understanding of the text, the label value of each label title cell is obtained; Based on the matching degree between the image tag value and the text in the image tag title cell, a final image tag value is determined, and the image tag title cell corresponding to the final image tag value is determined to obtain the image tag information.

7. The method for intelligently identifying frame information based on deep learning according to claim 5, characterized in that: The step of determining the label information in the target frame range based on the label title bar cell and the text bounding box includes: If there is no label title bar cell whose confidence information exceeds the threshold, the Qwen_VL multimodal model is used to predict the text in the label title bar cell.

8. A frame information intelligent recognition system based on deep learning, characterized in that: The system comprises: A frame range analysis module is used to extract the frame range under the layout and filter the frame range to obtain a target frame range; A graphic image analysis module, used to identify the label title bar cells in each target frame range based on a graphic recognition algorithm, and to predict the entire label bar area based on a yolov8 prediction model; The label information analysis module is used to determine the label information in the target frame range based on the label title bar cells and the text bounding box.

9. A terminal, characterized in that: The terminal includes a memory, a processor, and a deep learning-based intelligent frame information recognition program stored in the memory and executable on the processor. When the processor executes the deep learning-based intelligent frame information recognition program, the steps of the deep learning-based intelligent frame information recognition method as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a deep learning-based intelligent frame information recognition program. When the deep learning-based intelligent frame information recognition program is executed by the processor, the steps of the deep learning-based intelligent frame information recognition method as described in any one of claims 1 to 7 are implemented.

Images