Intelligent interaction method, device, equipment and medium based on digital projection lamp

Abstract

The present invention relates to the field of digital projection technology, and discloses an intelligent interaction method, device, equipment and medium based on a digital projection lamp. An input image is acquired by a high-definition camera; the input image is preprocessed; a target detection is performed on the input image by a YOLO target detection algorithm to identify a content area and an interactive projection area in the input image; a layout area detection is performed on the content area by using a graphic layout detection model to identify content areas of different categories; a pixel control instruction of a micro light emitting diode Micro-LED is generated according to the category of the content area and the interactive projection area; according to the pixel control instruction, a pixel lighting area of ​​the Micro-LED is controlled to generate an interactive graphic pattern and project it onto the interactive projection area, thereby realizing the intelligent interaction function of the digital projection lamp, being able to adapt to changes in the environment and the projection surface for identification and intelligent adjustment, and effectively improving and enriching the interactive experience.

Description

Technical Field

[0001] The present invention relates to the field of digital projection technology, and in particular to an intelligent interaction method, device, equipment and medium based on a digital projection lamp. Background Art

[0002] Currently, digital projection technology is widely used in education, exhibitions, entertainment and other scenarios. It is usually used to display static or dynamic content to facilitate users to obtain information. However, existing digital projection systems are mostly based on static projection and lack flexible interactive capabilities. Users can only passively watch the projected content and cannot interact with it in real time, which limits the application potential of projection technology. For example, in educational scenarios, teachers cannot dynamically adjust the projected content according to different teaching contents or the real-time situation of classroom interaction, and students cannot obtain personalized feedback on the projected content through direct interactive operations.

[0003] Some advanced digital projection systems have tried to introduce interactive functions, but most of them have only achieved simple gesture recognition or touch feedback, and the interactivity is still relatively limited. These systems are often unable to intelligently identify content areas, nor can they make personalized projection adjustments based on the actual content. In complex document layouts (such as those containing titles, text, tables, pictures, etc.), it is difficult for existing systems to distinguish different types of content and generate different interactive effects for each type of content. In addition, traditional projection technology lacks precise control over content details during interactive projection. For example, it is difficult to perform high-precision projection and dynamic response in specific areas of tables or pictures, and the interaction is not rich enough. Summary of the invention

[0004] In view of this, the present invention provides an intelligent interaction method based on a digital projection lamp to solve the technical problem that the digital projection interaction in the prior art is not rich enough.

[0005] According to a first aspect of the present invention, there is provided an intelligent interaction method based on a digital projection lamp, comprising:

[0006] Acquire input images through high-definition camera acquisition;

[0007] Preprocessing the input image, wherein the preprocessing includes graying, noise removal, edge detection and perspective transformation;

[0008] The YOLO target detection algorithm is used to detect the target of the input image and identify the content area and interactive projection area in the input image;

[0009] Performing layout area detection on the content area using the graphic layout detection model to identify different types of content areas, where the types of content areas include titles, texts, tables, pictures, formulas, and table / figure titles;

[0010] Generate pixel control instructions for Micro-LEDs according to the category of the content area and the interactive projection area;

[0011] According to the pixel control instructions, the pixel lighting area of ​​the Micro-LED is controlled to generate an interactive graphic pattern and project it onto the interactive projection area.

[0012] According to a second aspect of the present invention, there is provided an intelligent interactive device based on a digital projection lamp, comprising:

[0013] The acquisition module is used to acquire input images through a high-definition camera;

[0014] A preprocessing module, used for preprocessing the input image, wherein the preprocessing includes graying, noise removal, edge detection and perspective transformation;

[0015] The first recognition module is used to perform target detection on the input image by using the YOLO target detection algorithm to identify the content area and the interactive projection area in the input image;

[0016] A second recognition module is used to perform layout area detection on the content area using the graphic layout detection model to identify different types of content areas, wherein the types of content areas include titles, texts, tables, pictures, formulas, and table / figure titles;

[0017] A control module, used to generate pixel control instructions of a micro-light emitting diode Micro-LED according to the category of the content area and the interactive projection area;

[0018] The driving module is used to control the pixel lighting area of ​​the Micro-LED according to the pixel control instruction to generate an interactive graphic pattern and project it onto the interactive projection area.

[0019] According to a third aspect of the present invention, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned intelligent interaction method based on a digital projection lamp when executing the computer program.

[0020] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned intelligent interaction method based on a digital projection lamp are implemented.

[0021] Through the above technical scheme, the present invention provides an intelligent interaction method, device, equipment and medium based on a digital projection lamp, which acquires an input image through a high-definition camera; pre-processes the input image; performs target detection on the input image through the YOLO target detection algorithm, and identifies the content area and the interactive projection area in the input image; uses a graphic layout detection model to perform layout area detection on the content area to identify content areas of different categories; generates pixel control instructions for a micro light emitting diode Micro-LED according to the category of the content area and the interactive projection area; controls the pixel lighting area of ​​the Micro-LED according to the pixel control instructions to generate an interactive graphic pattern and project it onto the interactive projection area, thereby realizing the intelligent interaction function of the digital projection lamp, and can adapt to changes in the environment and the projection surface for identification and intelligent adjustment, effectively improving and enriching the interactive experience.

[0022] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention and implement it according to the contents of the specification, and to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0024] Figure 1 A schematic diagram showing an intelligent interactive application scenario of a digital projection lamp provided in an embodiment of the present invention;

[0025] Figure 2 A schematic flow chart of an intelligent interaction method based on a digital projection lamp provided in an embodiment of the present invention is shown;

[0026] Figure 3 A schematic diagram of a process flow of a data dimensionality reduction method provided in an embodiment of the present invention is shown;

[0027] Figure 4 A schematic diagram of a data enhancement process provided in an embodiment of the present invention is shown;

[0028] Figure 5 A schematic diagram of the training process of the image and text layout detection model provided in an embodiment of the present invention is shown;

[0029] Figure 6 A schematic structural diagram of an intelligent interactive device based on a digital projection lamp provided in an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0030] The specific implementation of the present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that the embodiments and features in the embodiments of the present invention can be combined with each other without conflict.

[0031] An intelligent interaction method based on a digital projection lamp provided by an embodiment of the present invention can be applied to Figure 1 In the scenario shown, the text content on the desktop (such as test papers, books, newspapers, etc.) is captured in real time by a high-definition camera, and the input image is preprocessed; the camera is connected to the controller to detect the target of the input image through the cloud model (YOLO target detection algorithm), and the content area and interactive projection area in the input image are identified; the text layout detection model is used to detect the layout area of ​​the text block content to identify different types of content areas (text blocks), where the types of content areas include titles, text, tables, pictures, formulas, and table / figure titles; according to the category of the content area and the interactive projection area, the pixel control instructions of the micro-light-emitting diode Micro-LED are generated; according to the pixel control instructions, the pixel lighting area of ​​the Micro-LED is controlled to generate an interactive graphic pattern and project it onto the projection area. The control system generates the corresponding Micro-LED pixel control instructions based on the identified content area category and the predefined interactive logic. For example, if the text area of ​​the book is detected, the system can project relevant annotations in the blank area next to it; if it is a test paper, the answer solution process or prompt information can be displayed in the blank area. Combined with actual application scenarios, such as classroom teaching or self-study scenarios, the system can recognize the test paper on the desktop and dynamically display the solution steps or annotations of the answer in the blank projection area to help students better understand the questions; in book reading scenarios, the system can display related annotations or notes in the blank area next to it, so that readers can have a richer reading experience; when reading news or newspapers, extended content or background information can be provided in the blank area to provide users with a deeper understanding of the information.

[0032] Embodiment 1:

[0033] like Figure 2 As shown, an intelligent interaction method based on a digital projection lamp provided in an embodiment of the present invention includes:

[0034] Step 201: acquiring an input image through a high-definition camera;

[0035] Step 202: preprocess the input image;

[0036] Among them, preprocessing includes grayscale, noise removal, edge detection and perspective transformation;

[0037] Step 203: Perform target detection on the input image using the YOLO target detection algorithm to identify the content area and the interactive projection area in the input image;

[0038] Wherein, step 203 specifically includes:

[0039] Step 203-1: Use the YOLO object detection algorithm to identify the content area in the input image, and generate a bounding box of the content area based on the edge detection result;

[0040] Step 203 - 2 : Calculate the interactive projection area based on the bounding box expansion of the content area, and identify the blank area around the content area as the interactive projection area.

[0041] Step 204: Perform layout area detection on the content area using the graphic layout detection model to identify different types of content areas;

[0042] The types of content areas include titles, text, tables, pictures, formulas, and table / figure titles. Before step 204, a training step of a graphic layout detection model is also included. The training of the graphic layout detection model specifically includes:

[0043] Step 204-1, generating a labeled data set based on content layouts of multiple content area types;

[0044] Step 204-2: performing data enhancement on the labeled data set to expand the data set;

[0045] Step 204-3: input the data set into the image-text layout detection model for training, and calculate the loss function;

[0046] Step 204-4: When the calculation result of the loss function is less than a preset threshold or the number of iterations is greater than the preset threshold, the model training is stopped and the graphic layout detection model is output; when the calculation result of the loss function is greater than the preset threshold and the number of iterations is less than the preset threshold, the parameters of the graphic layout detection model are optimized through back propagation.

[0047] Step 205: Generate pixel control instructions for the micro-light emitting diode Micro-LED according to the category of the content area and the interactive projection area;

[0048] Step 206: According to the pixel control instruction, the pixel lighting area of ​​the Micro-LED is controlled to generate an interactive graphic pattern and project it onto the interactive projection area.

[0049] An intelligent interaction method based on a digital projection lamp provided by an embodiment of the present invention acquires an input image through a high-definition camera; pre-processes the input image; performs target detection on the input image through a YOLO target detection algorithm to identify the content area and the interactive projection area in the input image; performs layout area detection on the content area using a graphic layout detection model to identify content areas of different categories; generates pixel control instructions for a micro light emitting diode Micro-LED according to the category of the content area and the interactive projection area; controls the pixel lighting area of ​​the Micro-LED according to the pixel control instructions to generate an interactive graphic pattern and project it onto the interactive projection area, thereby realizing the intelligent interaction function of the digital projection lamp, being able to adapt to changes in the environment and the projection surface for identification and intelligent adjustment, and effectively improving and enriching the interactive experience.

[0050] Embodiment 2:

[0051] In order to reduce the redundant features entering the model, the second embodiment of the present invention uses principal component analysis PCA to select fewer principal components to represent the main information of the original data, effectively reducing the data dimension, so that the model can achieve similar recognition effects at a lower computational complexity, thereby significantly improving the processing speed and real-time performance. Before step 204-3, a data dimensionality reduction process is introduced, as shown in FIG2, including:

[0052] Steps 204-301: Convert each image in the data set into a one-dimensional vector to form a data matrix ;

[0053] Among them, the matrix Each row represents the data of one image;

[0054] Steps 204-302, Calculate data matrix The mean vector m of the data point is subtracted from m;

[0055] Steps 204-303, calculating the covariance matrix;

[0056] Among them, the covariance matrix , n is the number of samples;

[0057] Steps 204-304, by covariance matrix Perform eigenvalue decomposition to obtain the covariance matrix The eigenvalues ​​and corresponding eigenvectors of

[0058] Steps 204-305: Sort in descending order according to the eigenvalues, and select the first The eigenvectors form a dimensionality reduction matrix , the data matrix after dimensionality reduction .

[0059] In the second embodiment of the present invention, PCA is used to perform dimensionality reduction processing on the collected image data before network input, thereby reducing data redundancy and improving training and reasoning speed. PCA retains useful features and reduces the participation of irrelevant features, thereby effectively reducing computing delays and improving the stability of the model. The dimensionality reduction strategy of PCA in this embodiment can effectively remove redundant background information in images or documents (the background of the environment in which the document is located, and the background of the carrier in which the text is located), and retain the core features of the text. For text recognition scenarios, PCA can be used to distinguish the key features of text and background, and eliminate noise and irrelevant background. This method is suitable for situations with complex document backgrounds, such as paper with watermarks or patterns, and can significantly improve the clarity and recognizability of the text.

[0060] Embodiment three:

[0061] In order to enhance the adaptability of the system to different document types and make it more flexible and efficient in practical applications, a data enhancement method is provided in the third embodiment of the present invention to realize the whole process from element extraction, sample selection, grid layout to optimized filling. The method makes the final document layout more compact and reasonable and conforms to the typesetting specification by accurately calculating the adaptability, filling rate and spatial overlap between elements and grids. Accordingly, Figure 4 As shown, step 204-2 specifically includes:

[0062] Step a: extract document elements of different content area types from the annotated dataset, construct an element pool, and group the elements according to element type and size;

[0063] Among them, all content areas of the document are extracted from the annotated dataset. These areas can include titles, text, images, tables, charts, formulas, etc. These areas are automatically identified and classified using image processing or optical character recognition (OCR) technology. Each content area extracted from the document is considered as an "element" and added to a pool. The elements in the pool include document components of various sizes and types, such as paragraphs, titles, pictures, etc. of different sizes. Elements are grouped by type (such as text, pictures, tables, etc.) and size (such as small blocks, headers, titles). The grouping of size and type can help subsequent steps to make a more reasonable layout when generating documents. This step can process each type of document element more targeted through classification and grouping, which facilitates element matching during subsequent optimization of layout. Ensure that the document components in the element pool are diverse and representative, and enhance the generalization ability of the model when processing different document formats.

[0064] Step b, selecting candidate samples from the element pool by stratified sampling, wherein each blank page generates a candidate sample set of different sizes;

[0065] Among them, in order to reasonably distribute each type of elements in the layout, stratified sampling technology is used to select samples from the element pool. Stratified sampling is based on the size and type of elements. A certain number of elements are selected in each layer as candidate samples, which can ensure that elements of each type and size in the sample set are representative. For each blank page (or page segment), candidate sample sets of different sizes are generated according to the actual size of the page. This means that the blank area on the page and the size of the elements will affect the selection of candidate samples, thereby providing a variety of choices for subsequent layouts. The stratified sampling in this step ensures that document elements of various types and sizes can be fully considered when generating the layout. By generating candidate sample sets based on the page size, the placement of elements can be optimized for different page structures, improving the rationality and effect of the layout.

[0066] Step c: based on the page size and document typesetting habits, divide the page into grids and generate a grid layout;

[0067] Among them, the analysis of page size and typesetting habits: according to the actual size of the page (such as A4, Letter and other paper sizes) and the typesetting habits of the document (such as left and right margins, paragraph indents, etc.), the page is divided into a uniform grid structure. The size and number of grids can be adjusted according to the content of the specific page. Each grid can accommodate elements of a certain size (such as a small block of text or image). Through the grid layout method, the content area of ​​the page is divided into several small units to provide a framework for the subsequent element placement. The grid layout provides a structured framework for the reasonable arrangement of document elements, avoiding excessive crowding or overlapping of elements, improving the accuracy of the layout process, and enabling elements to be optimally placed according to the actual size of the page and typesetting habits.

[0068] Step d: for each grid in the grid layout, traverse the elements in the element pool, calculate the filling rate of the elements and the grid, and preferentially select the element with the highest filling rate to place in the grid;

[0069] Among them, for each grid, all elements in the element pool are traversed, and the filling rate of each element and the grid is calculated. The filling rate can be calculated by the degree of overlap between the element and the grid, and the ratio of the size of the element to the size of the grid. The higher the filling rate, the more suitable the element is to be placed in the grid. Based on the calculation result of the filling rate, the element with the highest filling rate is preferentially selected to be placed in the grid. This ensures that the filling effect of each grid is optimal and avoids wasting space. By calculating the filling rate, this step can effectively optimize the space usage of elements, avoid unnecessary blanks or overcrowding, and ensure that the elements placed in each grid match the grid as much as possible, thereby improving the compactness and aesthetics of the document layout.

[0070] Step e: Check the affected grids in the layout, remove the grids that overlap or partially overlap with the placed elements and mark them as invalid, update the grid layout, and jump to step d to continue execution;

[0071] Among them, check the affected grids: in step e, if an element is placed in a grid, it will affect the layout of the surrounding grids (such as overlap or partial overlap), therefore, it is necessary to check the grids that overlap with the placed elements and mark them as invalid. For the grids marked as invalid, they need to be excluded from the candidate layout and the layout of other grids needs to be recalculated. Then, return to step d to continue optimizing the layout until the elements of all grids are placed reasonably. This checking mechanism in this step ensures that the elements in the placement process will not interfere with or overlap each other, improves the quality and neatness of the document layout, and dynamically adjusts the grid layout, which helps to improve the adaptability of the model and more effectively avoid layout conflicts when processing various complex documents.

[0072] Step f: When it is determined that all grids in the grid layout are filled with elements, the execution is terminated and the current annotated layout is added to the extended data set.

[0073] Among them, when all grids in the grid layout have been filled with elements (that is, there are no empty grids), the layout process stops. At this time, the currently generated document layout is considered valid, and subsequent operations can be performed to add the currently generated layout to the extended dataset. This layout can be used as training data to further optimize the performance of the model. This step ensures that no grids are missed during the layout process, avoiding invalid layout generation. By continuously expanding the dataset and adding diverse layout instances, more diverse data support is provided for the training of the document analysis model.

[0074] Embodiment 4:

[0075] In order to improve the accuracy of anchor box matching and layout detection, and enhance the adaptability to various document types and element layouts, such as Figure 5 As shown, step 204-3 specifically includes:

[0076] Step 204-31, based on the statistical results of the document elements in the data set, initializing anchor boxes of multiple scales at the beginning of model training;

[0077] Among them, each anchor box is assigned a different size according to the element type and size to cover various common objects in the document; the size and type of the elements (such as title, text, picture, table, formula, etc.) of each document in the training data set are analyzed. The size, aspect ratio, shape characteristics, etc. of each element are counted, and based on the above statistical results, multiple scale anchor boxes are initialized at the beginning of model training. The size and ratio of each anchor box are assigned according to the characteristics of common elements in the training data. For example, if some elements are large images, the anchor box will be larger; if it is a small text paragraph or formula, the anchor box will be smaller. In this way, the initialization of the anchor box covers various possible objects in the document. At initialization, the scale and ratio of the anchor box should be able to cover the diversity and variability of the document elements. For example, for horizontal long strip elements, the anchor box may have a higher width ratio; for rectangular or square elements, the aspect ratio of the anchor box tends to 1:1. This step initializes multiple scale anchor boxes according to the type and size of the document elements to ensure that various common elements in the document are covered. This provides richer candidate boxes for subsequent object detection, enabling the model to more comprehensively identify and locate different types of document elements.

[0078] Step 204-32: During the model training process, the anchor frame is dynamically adjusted according to the characteristics of different document elements to optimize the matching degree of the anchor frame;

[0079] Specifically, it includes using the intersection-over-union (IoU) threshold to determine the match between the anchor box and the true bounding box, and fine-tuning the size and proportion of the anchor box in real time according to the distribution of IoU to make it more in line with the actual detection needs. If the IoU of a certain anchor box has a low match with the true bounding box, the model will dynamically adjust the anchor box. According to the distribution of IoU, the model will adjust the size and aspect ratio of the anchor box in real time. For example, if the width of the anchor box is too large and the actual element is small, the model will adjust the width of the anchor box to improve the match. If the elements in some areas are large or small, the proportion and scale of the anchor box will be dynamically scaled to make the anchor box more suitable for the layout of the actual elements. This dynamic adjustment process is real-time. According to the calculation results of each IoU, the model will continuously adjust the parameters of the anchor box until the optimal match is achieved. This step can greatly improve the match between the anchor box and the true bounding box by adjusting the size and proportion of the anchor box in real time, reduce the number of unmatched anchor boxes, and improve the detection accuracy of the model, especially when facing the diversity and changes of document elements.

[0080] Step 204-33, calculate the loss function, and adjust the model weights according to the loss function, and fine-tune the anchor frame, and iterate repeatedly to gradually optimize it to be close to the real frame.

[0081] Among them, during the model training process, the calculation of the loss function involves two parts, namely positioning loss and classification loss. The positioning loss is mainly the deviation between the anchor box and the true bounding box. Usually, a smooth L1 loss or other loss functions suitable for regression problems are used. The goal of the loss function is to make the center position, width and height of the anchor box match the true bounding box as much as possible. Classification loss For each anchor box, the model also needs to determine whether the box contains a certain type of element (such as title, text, etc.). The classification loss usually uses the cross entropy loss function. In each round of iteration, according to the calculated loss value, the model will back-propagate and adjust the weight of the network to reduce the loss. In this way, the model gradually learns how to optimize the matching of the anchor box and improve its accuracy. In addition to adjusting the model weights, the parameters of the anchor box (such as size, aspect ratio) will also be fine-tuned according to the calculation results of the loss function. The optimized anchor box will match the document elements more accurately, further improving the detection effect. The calculation of the loss function and the optimization of the model in this step enable each anchor box to fit the real bounding box more accurately, thereby improving the accuracy of layout detection. Through multiple rounds of training, the model is continuously optimized, and the matching degree of the anchor box gradually approaches the real box, and finally a high-precision target detection effect can be achieved.

[0082] Embodiment 4 of the present invention combines anchor box initialization, dynamic adjustment and loss function optimization to form an efficient and accurate image and text layout detection training process. During the training process, the model not only improves the detection accuracy, but also enhances the adaptability to various document types and element layouts through precise anchor box matching and weight adjustment.

[0083] The structure of the graphic layout detection model in the embodiment of the present invention includes an input module, a controllable perception module, a feature fusion module, a global to local module, a feature output module and a segmentation and classification module, wherein the input module inputs the entire document image into the model and extracts the first feature through a convolutional layer; the controllable perception module obtains the first feature and obtains second features of different granularities through a multi-scale feature extractor; the feature fusion module is used to fuse different second features to generate a third feature; the global to local module obtains the third feature, extracts the global feature of the page, divides the page into several sub-regions and extracts the features of each sub-region, and obtains local detail features; the feature output module is used to output the global features, character sub-region features and detail features; the segmentation and classification module segments different layout elements in the document image through a segmentation network according to the global features, character sub-region features and detail features, and uses a classification network to predict the type of each segmented element region.

[0084] The input module receives the entire document image, usually a color image or a grayscale image. The image may contain various document elements, such as text, pictures, tables, etc. The input image is first processed through several convolutional layers, which extract low-level features (such as edges, textures, colors, etc.) from the original image. These features provide the basis for subsequent image processing, usually including simple spatial information and texture patterns. Through the action of the convolutional layer, the model can automatically extract useful low-level features from the image, laying the foundation for subsequent deep feature extraction. The input module ensures that all key content of the document image is captured so that the model can understand the visual structure of the document.

[0085] The controllable perception module starts with the first feature extracted by the input module, and the multi-scale feature extractor uses a multi-scale feature extractor to extract features of different granularities. This means that for each input image, the model is able to perceive the image content at multiple scales. For example, for small font text and large title text in a document, the model can extract features at different scales to adapt to the size differences of various elements. Through multi-scale feature extraction, the model can process document elements of different sizes at the same time, enhancing the model's expressiveness in complex document structures. This module helps capture various detailed features in the document, especially for diverse elements such as images, tables, formulas, etc., and can effectively distinguish different types of regions.

[0086] The second features of different granularities extracted in the controllable perception module will be processed by the feature fusion module. This module combines features of different scales through weighted fusion, splicing or other methods to generate a richer third feature. This fusion helps to integrate information from various scales and improve the recognition ability of the model. Feature fusion can retain important information from different scales, allowing the model to capture more comprehensive document layout features. By fusing features from different scales, the model can obtain a comprehensive information representation, reduce information loss, and improve the accuracy of document understanding.

[0087] Global feature extraction first extracts global features from the third feature to capture the overall layout and structure of the entire document. These global features can help the model understand the overall structure of the document, such as the approximate location and arrangement of paragraphs, titles, pictures, etc. on the page. The page is then divided into several sub-regions. For example, it can be divided into headers, footers, body text, image areas, etc. The features in each sub-region are further extracted to obtain detailed features. At this point, the perception of local details is more accurate, which helps to further subdivide the layout of the document. Through the combination of global and local features, the model can simultaneously understand the macro structure of the document (such as the layout of paragraphs) and micro details (such as text or images in each paragraph). This dual perception ability greatly improves the model's comprehensive understanding of the document layout. By dividing into sub-regions and extracting features for each region, the model can fine-tune the layout analysis for different regions, thereby improving the detection accuracy at the detail level.

[0088] The feature output module outputs global features, sub-region features, and local detail features to provide rich information for subsequent segmentation and classification. These features will be further input into the segmentation and classification module for actual document layout element detection. This module effectively passes multi-level feature information to the segmentation and classification module to ensure that subsequent tasks can rely on complete global and local information for judgment. The output multi-level features help the model have a multi-dimensional basis for segmentation and classification, thereby improving the accuracy and robustness of document layout.

[0089] The segmentation and classification module is responsible for segmenting different layout elements in the input image through the segmentation network. The goal is to separate the various regions in the document (such as text, title, image, table, etc.) from the entire page. This step is the basic region division, and the purpose is to provide clear region boundaries for subsequent classification. The segmented regions are input into the classification network, and the classification network predicts the type of each region. The classification network can determine whether a region is a different layout element such as a title, text, picture or table. The segmentation network ensures that each region of the document can be accurately segmented to avoid overlap or missed detection between different elements. The classification network ensures that the type of each segmented region is accurately predicted so that each layout element of the document can be correctly marked. By combining segmentation and classification, the model can complete a comprehensive analysis of the document layout, identify different types of elements and accurately locate their positions on the page.

[0090] The embodiment of the present invention adopts the structural design of the graphic layout detection model, and the graphic layout detection model can understand the document at multiple levels. The model can process the global layout and detailed content of the document at the same time, effectively improving the accuracy of document structure analysis. Through multi-scale perception and feature fusion, the model can extract document information at different granularities, enhancing the adaptability to complex document content. The segmentation and classification modules ensure the accurate identification of each area of ​​the document, and provide a clear structure for document analysis. This design in the embodiment of the present invention enables the model to efficiently and accurately identify various layout elements in the document, especially when facing diverse and complex documents, while still maintaining high detection accuracy and robustness.

[0091] Embodiment five:

[0092] Taking the application in the field of education, such as the interactive scene of test questions and question-answering questions as an example, the fifth embodiment of the present invention is used for interactive display of paper test questions and reference answers, and automatically projects the reference answers next to the paper questions, providing a convenient and intuitive learning and question-answering method. The specific steps include:

[0093] Step 501: Image acquisition and preprocessing

[0094] Among them, a high-definition camera is used to obtain images of paper test questions. The camera can be set at an appropriate angle to clearly capture the question area. The obtained image is preprocessed, including grayscale, noise removal, edge detection, etc., to improve the accuracy of subsequent target detection.

[0095] Step 502: Target detection and region recognition

[0096] Among them, the YOLO target detection algorithm is used to detect the target of the input image. The algorithm can identify the problem area and the interaction area in the image. For example, YOLO can help identify the question text, question number and surrounding blank area. Based on the detection results of YOLO, the content area is further detected by the graphic layout detection model to identify the question content (such as text, pictures, tables, etc.) and the blank area that can be used to project the answer.

[0097] Step 503: Extraction and preprocessing of reference answers

[0098] Among them, according to the type of question, the paper reference answer is image preprocessed to extract the answer part related to the question. If the reference answer contains multiple steps or sub-questions, it can be further segmented to extract key information. Optionally, if there is an answer database stored in the cloud, the reference answer matching the current question can be obtained from the reference answer database. Among them, the paper reference answer is image processed, and the step of extracting the answer part related to the question involves matching the answer with the question at the same time. The traditional method includes matching according to the question number, and the existing technology method often leads to wrong judgment due to the confusion of numbers, especially math problems. Therefore, the answer matching process and answer extraction are correspondingly optimized in the fifth embodiment of the present invention. Step 503 specifically includes:

[0099] Step 503-1, reference answer matching and extraction;

[0100] In the reference answer database, relevant answers are searched according to the question number or keywords. If the question number exists, the answer matching the number is directly extracted. If there are only keywords, the similarity between the question and the answer is calculated through a semantic similarity algorithm (such as BERT or Siamese network), and the answer with the highest similarity is selected from the database as the matching result.

[0101] Step 503-2, cutting out reference answers;

[0102] Among them, segmentation or block technology is used to divide the answer into multiple parts (such as steps, sub-questions) according to the logical structure, and only the parts related to the question are retained in combination with the characteristics of the question. For example: if the question requires solution steps, all processes are extracted; if the question only requires the final answer, the conclusion part is extracted, and the extracted answer is stored in the form of an image for easy projection.

[0103] Step 504: Answer projection

[0104] Among them, a digital projector is used to project the extracted reference answers next to the paper questions according to the interactive area identified in the previous step. The projector controls the pixels through micro-light-emitting diodes (Micro-LEDs) to generate clear interactive graphic patterns and project them to the area preset by the user. If the question area changes (for example, the paper test question is flipped or moved), the system can redefine the interactive area through real-time detection and image processing algorithms to ensure that the reference answers are always accurately projected to the correct location. By identifying the interactive method, users can choose to view specific problem-solving steps, formula derivations, or other related reference content.

[0105] This fifth embodiment can efficiently identify the content of paper test questions through image processing and layout area recognition, and project relevant reference answers next to them. Through this intelligent interactive display, users can more conveniently obtain problem-solving ideas and answers, improve learning efficiency, and is particularly suitable for automatic question answering and learning tutoring scenarios. In addition, the system's adaptive ability can cope with various environmental changes to ensure that the projected content is always accurate.

[0106] To achieve the above functions, the interactive system includes a high-definition camera for capturing images of paper test questions, including questions and related areas; the YOLO target detection algorithm in the example of the present invention is run in the processor to detect and identify content areas (such as question descriptions, blank areas, etc.) and interactive projection areas in the image; the image processing module is run in the processor to process the captured image, such as cropping, graying, noise removal, etc., so as to subsequently extract the answer area; the digital projector projects the extracted reference answer next to the paper question based on the identified interactive area.

[0107] Further, as Figure 2 The specific implementation of the method, the embodiment of the present invention provides an intelligent interactive device based on a digital projection lamp, such as Figure 6 As shown, the device comprises:

[0108] The acquisition module 610 is used to acquire input images through a high-definition camera;

[0109] A preprocessing module 620, used to preprocess the input image, wherein the preprocessing includes graying, noise removal, edge detection and perspective transformation;

[0110] A first recognition module 630 is used to perform target detection on the input image by using the YOLO target detection algorithm to identify the content area and the interactive projection area in the input image;

[0111] The second identification module 640 is used to perform layout area detection on the content area using the graphic layout detection model to identify different types of content areas, wherein the types of content areas include titles, texts, tables, pictures, formulas, and table / figure titles;

[0112] The control module 650 is used to generate a pixel control instruction of the micro-light emitting diode Micro-LED according to the category of the content area and the interactive projection area;

[0113] The driving module 660 is used to control the pixel lighting area of ​​the Micro-LED according to the pixel control instruction to generate an interactive graphic pattern and project it onto the interactive projection area.

[0114] In an embodiment of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0115] Acquire input images through high-definition camera acquisition;

[0116] Preprocessing the input image, wherein the preprocessing includes graying, noise removal, edge detection and perspective transformation;

[0117] The YOLO target detection algorithm is used to detect the target of the input image and identify the content area and interactive projection area in the input image;

[0118] Performing layout area detection on the content area using the graphic layout detection model to identify different types of content areas, where the types of content areas include titles, texts, tables, pictures, formulas, and table / figure titles;

[0119] Generate pixel control instructions for Micro-LEDs according to the category of the content area and the interactive projection area;

[0120] According to the pixel control instructions, the pixel lighting area of ​​the Micro-LED is controlled to generate an interactive graphic pattern and project it onto the interactive projection area.

[0121] It should be noted that the above embodiments only use the recognition interaction of common paper documents to illustrate the principles and implementation steps of the embodiments of the present invention, and do not specifically limit the actual application scenarios. For the functions or steps that can be implemented by computer-readable storage media or computer devices, please refer to the above-mentioned method embodiments. To avoid repetition, they will not be described one by one here.

[0122] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0123] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0124] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. An intelligent interaction method based on digital projection lamp, characterized in that: include: Acquire input images through high-definition camera acquisition; Preprocessing the input image, wherein the preprocessing includes graying, noise removal, edge detection and perspective transformation; Performing target detection on the input image by using the YOLO target detection algorithm to identify the content area and the interactive projection area in the input image; Training a graphic layout detection model, specifically including: generating annotated datasets based on content layouts of multiple content area types; performing data augmentation on the annotated datasets to expand the datasets; converting each image in the dataset into a one-dimensional vector to form a data matrix , where the matrix Each row represents the data of an image; calculate the data matrix The mean vector m of each data point is subtracted from m; calculate the covariance matrix , where n is the number of samples; by the covariance matrix Perform eigenvalue decomposition to obtain the covariance matrix The eigenvalues ​​and corresponding eigenvectors of The eigenvectors form a dimensionality reduction matrix , the data matrix after dimensionality reduction ; Input the data set into the image and text layout detection model for training, and calculate the loss function; when the loss function calculation result is less than a preset threshold or the number of iterations is greater than the preset threshold, stop the model training and output the image and text layout detection model; when the loss function calculation result is greater than the preset threshold and the number of iterations is less than the preset threshold, optimize the parameters of the image and text layout detection model through back propagation; Perform data enhancement on the annotated data set to expand the data set, the steps specifically include: step a: extract document elements of different content area types from the annotated data set, build an element pool, and group the elements according to element type and size; step b: select candidate samples from the element pool through stratified sampling, wherein each blank page generates a candidate sample set of different sizes; step c: based on the page size and document typesetting habits, divide the page into grids to generate a grid layout; step d: for each grid in the grid layout, traverse the elements in the element pool, and by calculating the filling rate of the element and the grid, preferentially select the element with the highest filling rate to place in the grid; step e: check the layout affected The affected grids are removed and marked as invalid if they overlap or partially overlap with the placed elements, the grid layout is updated, and the execution is skipped to step d for further execution; step f: when it is determined that all grids in the grid layout are filled with elements, the execution is terminated, and the current annotated layout is added to the extended data set, wherein the data set is input into the image and text layout detection model for training, and the step of calculating the loss function comprises: according to the statistical results of the document elements in the data set, anchor frames of multiple scales are initialized at the beginning of the model training, wherein each anchor frame is assigned a different size according to the element type and size to cover various common objects in the document; during the model training process, the anchor frame is dynamically adjusted according to the characteristics of different document elements to optimize the matching degree of the anchor frame, specifically including using the intersection-over-union (IoU) threshold to determine the matching between the anchor frame and the true bounding box, and fine-tuning the size and proportion of the anchor frame in real time according to the distribution of IoU to make it more in line with the actual detection requirements; the loss function is calculated, and according to the loss function, the model weight is adjusted, and the anchor frame is fine-tuned at the same time, and it is iterated repeatedly to gradually optimize it to be close to the true frame; Performing layout area detection on the content area using a graphic layout detection model to identify different types of content areas, wherein the types of the content area include titles, texts, tables, pictures, formulas, and table / figure titles; Generate pixel control instructions for Micro-LEDs according to the category of the content area and the interactive projection area; According to the pixel control instruction, the pixel lighting area of ​​the Micro-LED is controlled to generate an interactive graphic pattern and project it onto the interactive projection area.

2. The intelligent interaction method based on digital projection lamp according to claim 1, characterized in that: The step of performing target detection on the input image by using the YOLO target detection algorithm to identify the content area and the interactive projection area in the input image includes: Using the YOLO object detection algorithm to identify the content area in the input image, and based on the edge detection result, generating a bounding box of the content area; An interactive projection area is calculated based on the bounding box expansion of the content area, and a blank area around the content area is identified as the interactive projection area.

3. The intelligent interaction method based on digital projection lamp according to claim 1, characterized in that: The structure of the graphic layout detection model includes an input module, a controllable perception module, a feature fusion module, a global to local module, a feature output module and a segmentation and classification module, wherein the input module inputs the entire document image into the model and extracts the first feature through a convolutional layer; the controllable perception module obtains the first feature and obtains second features of different granularities through a multi-scale feature extractor; the feature fusion module is used to fuse different second features to generate a third feature; the global to local module obtains the third feature, extracts the global feature of the page, divides the page into several sub-regions and extracts the features of each sub-region, and obtains local detail features; the feature output module is used to output the global features, character sub-region features and detail features; the segmentation and classification module segments different layout elements in the document image through a segmentation network according to the global features, character sub-region features and detail features, and uses a classification network to predict the type of each segmented element region.

4. An intelligent interactive device based on a digital projection lamp, characterized in that: include: The acquisition module is used to acquire input images through a high-definition camera; A preprocessing module, used for preprocessing the input image, wherein the preprocessing includes graying, noise removal, edge detection and perspective transformation; A first recognition module is used to perform target detection on the input image by using the YOLO target detection algorithm to identify the content area and the interactive projection area in the input image; The second recognition module is used to perform layout area detection on the content area using a graphic layout detection model to identify different categories of content areas, wherein the types of the content areas include titles, texts, tables, pictures, formulas, and table / figure titles, wherein the steps of training the graphic layout detection model specifically include: generating an annotated data set based on the content layout of multiple content area types; performing data enhancement on the annotated data set to expand the data set; converting each image in the data set into a one-dimensional vector to form a data matrix , where the matrix Each row represents the data of an image; calculate the data matrix The mean vector m of each data point is subtracted from m; calculate the covariance matrix , where n is the number of samples; by the covariance matrix Perform eigenvalue decomposition to obtain the covariance matrix The eigenvalues ​​and corresponding eigenvectors of The eigenvectors form a dimensionality reduction matrix , the data matrix after dimensionality reduction ; Input the data set into the image and text layout detection model for training, and calculate the loss function; when the loss function calculation result is less than a preset threshold or the number of iterations is greater than the preset threshold, stop the model training and output the image and text layout detection model; when the loss function calculation result is greater than the preset threshold and the number of iterations is less than the preset threshold, optimize the parameters of the image and text layout detection model by back propagation; Perform data enhancement on the annotated data set to expand the data set, the step specifically includes step a: extract document elements of different content area types from the annotated data set, build an element pool, and group the elements according to element type and size; step b: select candidate samples from the element pool by stratified sampling, wherein each blank page generates a candidate sample set of different sizes; step c: based on the page size and document typesetting habits, divide the page into grids to generate a grid layout; step d: for each grid in the grid layout, traverse the elements in the element pool, and by calculating the fill rate of the element and the grid, preferentially select the element with the highest fill rate to place in the grid; step e: check the layout affected The affected grids are removed and marked as invalid if they overlap or partially overlap with the placed elements, the grid layout is updated, and the execution is skipped to step d for further execution; step f: when it is determined that all grids in the grid layout are filled with elements, the execution is terminated, and the current annotated layout is added to the extended data set, wherein the data set is input into the image and text layout detection model for training, and the step of calculating the loss function comprises: according to the statistical results of the document elements in the data set, anchor frames of multiple scales are initialized at the beginning of the model training, wherein each anchor frame is assigned a different size according to the element type and size to cover various common objects in the document; during the model training process, the anchor frame is dynamically adjusted according to the characteristics of different document elements to optimize the matching degree of the anchor frame, specifically including using the intersection-over-union (IoU) threshold to determine the matching between the anchor frame and the true bounding box, and fine-tuning the size and proportion of the anchor frame in real time according to the distribution of IoU to make it more in line with the actual detection requirements; the loss function is calculated, and according to the loss function, the model weight is adjusted, and the anchor frame is fine-tuned at the same time, and it is iterated repeatedly to gradually optimize it to be close to the true frame; A control module, used to generate pixel control instructions of a micro-light emitting diode Micro-LED according to the category of the content area and the interactive projection area; The driving module is used to control the pixel lighting area of ​​the Micro-LED according to the pixel control instruction to generate an interactive graphic pattern and project it onto the interactive projection area.

5. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the intelligent interaction method based on the digital projection lamp as claimed in any one of claims 1 to 3 are implemented.

6. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the intelligent interaction method based on a digital projection lamp as claimed in any one of claims 1 to 3 are implemented.

Images