Teaching schematic diagram visual positioning method and device based on adaptive format tower perception
Through the adaptive Gestalt perception method, the multimodal feature data set is constructed and the visual positioning model is trained, which solves the problem that traditional methods cannot handle the small sample characteristics of teaching schematic diagrams, and realizes accurate visual positioning under the limited sample size, supporting educational and teaching Multimodal retrieval and Q&A applications of scenarios.
Patent Information
- Application Number
- CN202411051785.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-01
- Publication Date
- 2024-11-29
AI Technical Summary
Traditional image representation methods have failed to effectively solve the visual semantic gap caused by small sample characteristics, sparse visual characteristics and complex semantic characteristics in teaching diagrams, and cannot be directly applied to visual positioning of teaching diagrams.
Using an adaptive Gestalt perception method, a multimodal feature data set is constructed by obtaining high-level semantic features, text description features and visual features of tiles of teaching diagrams, training the pre-constructed visual positioning model, and then filtering and deploying it. Visual positioning on electronic devices.
The visual representation ability of teaching diagrams is improved, ensuring accurate visual positioning is achieved when sample size is limited, and supporting multimodal retrieval and question-and-answer applications in educational and teaching scenarios.
Smart Images

Figure CN119049033A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of educational and teaching resource analysis, and specifically relates to a teaching schematic diagram visual positioning method and device based on adaptive Gestalt perception. Background Art
[0002] Teaching schematic visual localization aims to capture the association between language description and local objects in the schematic, and plays an important role in applications such as online teaching question answering and cross-modal retrieval. Current traditional image representation methods rely on deep learning models driven by large-scale annotated data, without comprehensively considering the small sample characteristics, sparse visual characteristics and complex semantic characteristics of teaching schematics. Traditional methods cannot be directly applied to teaching schematic representation.
[0003] Therefore, there is an urgent need for teaching schematic visual localization methods that can alleviate the small sample characteristics and the visual-semantic gap. Summary of the invention
[0004] In order to solve the above problems existing in the prior art, the present invention provides a teaching schematic visual positioning method and device based on adaptive Gestalt perception. The technical problem to be solved by the present invention is achieved by the following technical solutions:
[0005] In a first aspect, the present invention provides a teaching schematic visual positioning method based on adaptive Gestalt perception, comprising:
[0006] S100, obtaining a sample set including a plurality of teaching schematic diagrams;
[0007] S200, extracting high-level semantic features, text description features, and visual features of each teaching schematic diagram in the sample set, and constructing a multimodal feature dataset of the sample set using the high-level semantic features, visual features, and text description features;
[0008] S300, training a pre-built visual localization model using the multimodal feature data set to obtain a trained visual localization model;
[0009] S400, screening the trained visual positioning model, and after screening, deploying it on an electronic device to perform visual positioning on the positioning teaching diagram, and displaying it to the user.
[0010] In a second aspect, the present invention provides a teaching schematic visual positioning device based on adaptive Gestalt perception, comprising:
[0011] An acquisition module is configured to acquire a sample set including a plurality of teaching schematic diagrams;
[0012] An extraction module is configured to extract high-level semantic features, text description features and visual features of each teaching schematic diagram in the sample set, and construct a multimodal feature data set of the sample set using the high-level semantic features, visual features and text description features;
[0013] A training module is configured to train a pre-built visual positioning model using the multimodal feature data set to obtain a trained visual positioning model;
[0014] The positioning module is configured to screen the trained visual positioning model. After the screening is completed, it is deployed on the electronic device to perform visual positioning on the positioning teaching diagram and display it to the user.
[0015] Beneficial effects:
[0016] One or more technical solutions provided in the present invention have at least the following technical effects or advantages:
[0017] (1) The present invention adopts adaptive Gestalt perception as a priori, which actually introduces external knowledge that does not require additional annotation into the positioning model. The adaptive Gestalt perception prior does not rely on large-scale annotation information, and can efficiently identify meaningful visual areas in teaching schematics, thereby improving the model's visual representation ability for teaching schematics.
[0018] (2) The present invention designs a visual-text blender, which is guided by multimodal contextual attention and prompts the model to filter out content in the teaching diagram that is irrelevant to the target query, thereby constraining the target query to focus only on the visual area in the teaching diagram that is most similar to the text description semantics, thereby ensuring the accuracy of target query decoding.
[0019] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] Figure 1 It is a flow chart of a teaching diagram visual positioning method based on adaptive Gestalt perception provided by the present invention;
[0021] Figure 2 This is an example of annotation of a sample set of teaching schematic diagrams provided by the present invention;
[0022] Figure 3 It is a training schematic diagram of a teaching schematic diagram visual positioning model based on adaptive Gestalt perception provided by the present invention;
[0023] Figure 4 This is a comparison example of visual positioning results of the teaching schematic diagram provided by the present invention. DETAILED DESCRIPTION
[0024] The present invention is further described in detail below with reference to specific embodiments, but the embodiments of the present invention are not limited thereto.
[0025] like Figure 1 As shown, the present invention provides a teaching schematic diagram visual positioning method based on adaptive Gestalt perception, including:
[0026] S100, obtaining a sample set including a plurality of teaching schematic diagrams; wherein the object area in each teaching schematic diagram in the sample set is selected by a labeling frame, and each labeling frame has a corresponding description text; illustratively, referring to Figure 2 ,Firstly, any teaching schematic is selected and all the contained object regions are selected with a rectangular box, and the corresponding text description is ,labeled for each object region to make a sample set of teaching schematics, ,which contains 2038 teaching schematics and 132k text descriptions.
[0027] S200, extracting high-level semantic features, text description features, and visual features of each teaching schematic diagram in the sample set, and constructing a multimodal feature dataset of the sample set using the high-level semantic features, visual features, and text description features;
[0028] The visual features include color features, position features and edge features, and the text description features include text description word features.
[0029] For example, through the three-dimensional image samples, the teaching schematic features are extracted, such as the high-level semantic features F D , color features, position features, edge features, etc. of the blocks; extract text description word features through text description; then combine all features to build a multimodal feature dataset.
[0030] S300, training a pre-built visual localization model using the multimodal feature data set to obtain a trained visual localization model;
[0031] S400, screening the trained visual positioning model, and after screening, deploying it on an electronic device to perform visual positioning on the positioning teaching diagram, and displaying it to the user.
[0032] The present invention discloses a method and device for visual positioning of teaching diagrams based on adaptive Gestalt perception, which relates to the technical field of educational and teaching resource analysis, and solves the problem that traditional large-scale annotation data-driven technology cannot be directly applied because it does not comprehensively consider the small sample characteristics, sparse visual characteristics and complex semantic characteristics of teaching diagrams. The method includes: obtaining a sample set containing multiple teaching diagrams; then extracting high-level semantic features, text description features and visual features of blocks, and then constructing a multimodal feature data set; using the multimodal feature data set to train a pre-constructed visual positioning model to obtain a trained visual positioning model, and then screening it. After the screening is completed, it is deployed on an electronic device to visually locate the teaching diagram to be located and display it to the user. The present invention realizes the guarantee of the accuracy of visual positioning of teaching diagrams under the condition of limited sample size, and provides good technical support for multimodal retrieval and question-and-answer applications in educational and teaching scenarios.
[0033] refer to Figure 3 , refer to Figure 3 The schematic diagram of the process of constructing a visual positioning model of a teaching diagram based on adaptive Gestalt perception prior is shown in S300 of the present invention.
[0034] S310, defining an adaptive Gestalt perception factor of visual features in a visual localization model, a visual text blender in the visual localization model, a visual localization decoder in the visual localization model, and an optimization loss function of the visual localization model; wherein the visual features include: color features, position features, and tile edge features;
[0035] S320, in the current iteration, selecting high-level semantic features of a teaching schematic diagram, visual features of blocks in the teaching schematic diagram, and text description word features as input in the multimodal feature data set, and using the adaptive Gestalt perception factor of the visual features of each block to calculate the adaptive Gestalt perception features of the input teaching schematic diagram;
[0036] S330, inputting the high-level semantic features, the text description word features and the adaptive Gestalt perception features in S320 into the visual text blender to perform visual text blending to obtain fused features;
[0037] S340, inputting the fused features into the visual positioning decoder to perform visual positioning, and obtaining the spatial position coordinates of the object area corresponding to the description text of the teaching schematic diagram and marked with a prediction box;
[0038] S350, calculating the loss value of the input teaching schematic diagram using the optimization loss function according to the spatial position coordinates of the object area marked by the prediction box in S340 and the position coordinates of the object area marked by the marking box;
[0039] S360, using the loss value to adjust the weight parameters of the visual localization model, and reselecting high-level semantic features of a teaching schematic diagram, visual features of blocks in the teaching schematic diagram, and text description word features as input in the multimodal feature dataset;
[0040] S370, repeat S320 to S370 until the iteration end condition is reached to obtain a trained visual positioning model.
[0041] As a specific implementation of the present invention, the adaptive Gestalt perception factor defining the visual features in the visual localization model includes:
[0042] a, map the color moment of the block to the color feature space, map the position coordinates of the upper left vertex and the lower right vertex of the block to the position feature space, and map the boundary pixel value of the block to the edge feature space;
[0043] Wherein, the color feature is the color moment, the edge feature is the boundary pixel value, and the position feature is the position coordinate;
[0044] b, respectively using the color feature, position feature, and edge feature as nodes of three Gestalt perception subgraph networks, and the correlation between the features as edges to obtain three Gestalt perception subgraph networks;
[0045] This step initializes three Gestalt-aware subgraph networks, with block color features, position features, and edge features as nodes, and the correlation between features as edges to obtain three Gestalt-aware subgraph networks.
[0046] c. Using the corresponding features mapped to the color feature space, position feature space and edge feature space and the three Gestalt perception sub-image networks, the color similarity α, position proximity β and contour continuity γ between each pair of image blocks are calculated respectively;
[0047] d. Set the adaptive weights of color similarity, position proximity and contour continuity respectively, and then use the color similarity, position proximity and contour continuity and their corresponding adaptive weights for weighted summation to obtain the adaptive Gestalt perception factors of the color feature, the position feature and the edge feature.
[0048] As a specific implementation of the present invention, S320 includes:
[0049] S321, in the current iteration, selecting high-level semantic features of a teaching schematic diagram, visual features of blocks in the teaching schematic diagram, and text description word features as input in the multimodal feature dataset;
[0050] S322, for any two blocks in the input teaching schematic diagram, the adaptive Gestalt perception factors corresponding to the color features, position features and edge features of the arbitrary two blocks are compared with the probability threshold; if the adaptive Gestalt perception factor is greater than the probability threshold, the color features, position features and edge features of the arbitrary two blocks are correspondingly fused to obtain the adaptive Gestalt perception features of the input teaching schematic diagram.
[0051] It is worth noting that this step first sets the probability threshold θ for judging whether two blocks are from the same visual object. If the comprehensive adaptive Gestalt perception factor is greater than θ, it means that the two blocks have similar color features, their spatial positions on a teaching diagram are close, and the contours of the objects in the two blocks can be combined into a continuous curve. The probability that the two objects are combined into a complete object is high. The teaching diagram blocks that meet the judgment conditions of the comprehensive adaptive Gestalt perception factor are subjected to feature aggregation to obtain the teaching diagram adaptive Gestalt perception feature F generated by the adaptive Gestalt perception prior. G .
[0052] As a specific implementation of the present invention, S330 includes:
[0053] S331, defining the number of attention heads H and the mapping dimension d of the channel of the multi-head attention mechanism in the visual text blender;
[0054] S332, using the adaptive Gestalt perception features of the teaching schematic diagram as keys and values in a multi-head attention mechanism, and using the high-level semantic features of the teaching schematic diagram as a query;
[0055] S333, calculating the attention scores of the adaptive Gestalt perception features and the high-level semantic features, and using the attention to screen the most effective adaptive Gestalt perception features for the high-level semantic representation, and obtaining the teaching schematic representation of adaptive Gestalt perception enhancement by weighted summation, which is expressed as:
[0056]
[0057] Among them, W Q , W k , W V They represent the learnable linear mapping parameters of query, key and value respectively, softmax is the probability distribution calculated by row, and d represents the mapping dimension of the channel.
[0058] S334, using the text description word element feature as a key and value, and the high-level semantic feature of the teaching schematic diagram as a query, calculating the attention score of the text description word element feature and the high-level semantic feature of the teaching schematic diagram, using the attention to obtain the most effective text description word element feature for screening high-level semantic representation, and obtaining the teaching schematic diagram representation F enhanced by the text word element by weighted summation. DT ;
[0059] S335, taking the teaching schematic representation enhanced by adaptive Gestalt perception and the teaching schematic representation enhanced by text words as input, calculating the channel attention and spatial attention of the features, and performing weighted summation to obtain the fusion features of the semantically enhanced teaching schematic.
[0060] In order to refine the representation of teaching schematics, channel attention gating and spatial attention gating are defined to filter out irrelevant background noise in the schematics respectively. After filtering with the two gates, high-quality teaching schematic features with visual and text fusion are obtained.
[0061] As a specific implementation of the present invention, S340 includes:
[0062] S341, defining the number of attention heads H and feature mapping dimension of the multi-head attention mechanism in the visual localization decoder;
[0063] S342, using the fused features of the semantically enhanced teaching schematic diagram as the key and value in the attention mechanism, calculating the attention score of the target query and the fused features, and screening the target features of the teaching schematic diagram that are most relevant to the text description; wherein the target query is a predefined learnable parameter;
[0064] S343, decoding the most relevant target features of the target query into the spatial position coordinates of the object area corresponding to the description text on the teaching schematic diagram and marked with a prediction box.
[0065] This step decodes the corresponding prediction box (x b ,y b ,w b ,h b ), where (x b ,y b ) is the coordinate of the center point of the prediction box, (w b ,h b ) are the width and height of the prediction box.
[0066] As a specific implementation of the present invention, S350 includes:
[0067] The spatial position coordinates of the object area marked by the prediction box and the spatial position coordinates of the object area marked by the annotation box are input into the optimization loss function formula to calculate the loss value of the input teaching schematic diagram.
[0068] In this step, the prediction box b and the annotation box in the teaching diagram sample set are first obtained. The prediction result score uses the GIoU indicator, which is defined as:
[0069]
[0070] Among them, A c is the minimum closure area of the prediction box and the annotation box, and U is the union of the prediction box and the annotation box.
[0071] Based on the GIoU indicator, calculate the intersection-over-union loss function of the predicted box and the labeled box and mean absolute error loss in, Using the MAE function, the calculation method is defined as:
[0072]
[0073] Among them, A c is the minimum closure area of the prediction box and the annotation box, U is the union of the prediction box and the annotation box, and b i is the i-th prediction box in the teaching diagram, is the i-th annotation box of the teaching diagram, and N is the total number of teaching diagrams in the sample set.
[0074] The final optimized loss function is expressed as:
[0075]
[0076] Among them, λ giou and represents the weight balancing factor; b represents the spatial position coordinates of the object area marked by the prediction box; Represents the spatial position coordinates of the object area marked by the annotation box; represents the intersection-over-union loss function, represents the mean absolute error loss function.
[0077] As a specific implementation of the present invention, before S400, the teaching schematic diagram visual positioning method based on adaptive Gestalt perception also includes:
[0078] Acquire a test teaching schematic diagram, and use a trained visual positioning model to perform visual positioning on the test teaching schematic diagram to obtain the spatial position coordinates of the object area corresponding to the description text of the test teaching schematic diagram;
[0079] evaluating the performance of the visual localization model using the spatial position coordinates of the object area corresponding to the description text of the test teaching schematic diagram;
[0080] The visual localization model whose performance meets the requirements is used as the screened visual localization model.
[0081] In this embodiment, the teaching schematic visual positioning model based on adaptive Gestalt perception is compared with the single-stage zero-sample visual positioning model ZeroSG, the recursive visual positioning model RecursiveG and the multimodal verification reasoning model Visual-LinguisticG. Specifically, the obtained accuracy index is referred to Table 1. Table 1 is a comparison table of the performance evaluation effect of the visual positioning of the teaching schematic diagram. The comparative experimental results show that the positioning accuracy of the teaching schematic visual positioning model based on adaptive Gestalt perception on the test set can reach 0.2767, which is significantly higher than the single-stage zero-sample visual positioning model ZeroSG and the recursive visual positioning model RecursiveG, and is also better than the existing mainstream multimodal verification reasoning model Visual-LinguisticG.
[0082] Table 1 Comparison of experimental results
[0083]
[0084]
[0085] In addition, for example, please refer to Figure 4 , Figure 4 This is a comparison of the positioning results of the test set samples by the single-stage zero-shot visual localization model ZeroSG and the visual localization model based on adaptive Gestalt perception prior. The solid line represents the real annotation bounding box, the short horizontal line represents the predicted bounding box of the single-stage zero-shot visual localization model ZeroSG, and the dotted line represents the predicted bounding box based on the adaptive Gestalt perception prior visual localization model.
[0086] In a second aspect, the present invention provides a teaching schematic visual positioning device based on adaptive Gestalt perception, comprising:
[0087] An acquisition module is configured to acquire a sample set including a plurality of teaching schematic diagrams;
[0088] An extraction module is configured to extract high-level semantic features, text description features and visual features of each teaching schematic diagram in the sample set, and construct a multimodal feature data set of the sample set using the high-level semantic features, visual features and text description features;
[0089] A training module is configured to train a pre-built visual positioning model using the multimodal feature data set to obtain a trained visual positioning model;
[0090] The positioning module is configured to screen the trained visual positioning model. After the screening is completed, it is deployed on the electronic device to perform visual positioning on the positioning teaching diagram and display it to the user.
[0091] It is worth noting that the terms "first" and "second" in the present invention are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.
[0092] Although the present application is described herein in conjunction with various embodiments, in the process of implementing the claimed application, those skilled in the art may understand and implement other variations of the disclosed embodiments by reviewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality of components or steps.
[0093] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.
Claims
1. A teaching diagram visual positioning method based on adaptive Gestalt perception, characterized in that: include: S100, obtaining a sample set including a plurality of teaching schematic diagrams; S200, extracting high-level semantic features, text description features, and visual features of each teaching schematic diagram in the sample set, and constructing a multimodal feature dataset of the sample set using the high-level semantic features, visual features, and text description features; S300, training a pre-built visual localization model using the multimodal feature data set to obtain a trained visual localization model; S400, screening the trained visual positioning model, and after screening, deploying it on the electronic device to perform visual positioning on the positioning teaching diagram, and displaying it to the user.
2. The teaching diagram visual positioning method based on adaptive Gestalt perception according to claim 1 is characterized in that: The object area in each teaching diagram in the sample set is selected with a label box, and each label box has a corresponding description text; the visual features include color features, position features and edge features, and the text description features include text description word features.
3. The teaching diagram visual positioning method based on adaptive Gestalt perception according to claim 2 is characterized in that: S300 includes: S310, defining an adaptive Gestalt perception factor of visual features in a visual localization model, a visual text blender in the visual localization model, a visual localization decoder in the visual localization model, and an optimization loss function of the visual localization model; wherein the visual features include: color features, position features, and tile edge features; S320, in the current iteration, selecting high-level semantic features of a teaching schematic diagram, visual features of blocks in the teaching schematic diagram, and text description word features as input in the multimodal feature data set, and using the adaptive Gestalt perception factor of the visual features of each block to calculate the adaptive Gestalt perception features of the input teaching schematic diagram; S330, inputting the high-level semantic features, the text description word features and the adaptive Gestalt perception features in S320 into the visual text blender to perform visual text blending to obtain fused features; S340, inputting the fused features into the visual positioning decoder to perform visual positioning, and obtaining the spatial position coordinates of the object area corresponding to the description text of the teaching schematic diagram and marked with a prediction box; S350, calculating the loss value of the input teaching schematic diagram using the optimization loss function according to the spatial position coordinates of the object area marked by the prediction box in S340 and the position coordinates of the object area marked by the marking box; S360, using the loss value to adjust the weight parameters of the visual localization model, and reselecting high-level semantic features of a teaching schematic diagram, visual features of blocks in the teaching schematic diagram, and text description word features as input in the multimodal feature dataset; S370, repeat S320 to S370 until the iteration end condition is reached to obtain a trained visual positioning model.
4. The teaching diagram visual positioning method based on adaptive Gestalt perception according to claim 3 is characterized in that: The adaptive Gestalt perceptual factors that define visual features in the visual localization model include: a, map the color moment of the block to the color feature space, map the position coordinates of the upper left vertex and the lower right vertex of the block to the position feature space, and map the boundary pixel value of the block to the edge feature space; Wherein, the color feature is the color moment, the edge feature is the boundary pixel value, and the position feature is the position coordinate; b, respectively using the color feature, position feature, and edge feature as nodes of three Gestalt perception subgraph networks, and the correlations between the features as edges to obtain three Gestalt perception subgraph networks; c, using the corresponding features mapped to the color feature space, position feature space and edge feature space and the three Gestalt perception sub-image networks, the color similarity, position proximity and contour continuity between two blocks are calculated respectively; d. Set the adaptive weights of color similarity, position proximity and contour continuity respectively, and then use the color similarity, position proximity and contour continuity and their corresponding adaptive weights for weighted summation to obtain the adaptive Gestalt perception factors of the color feature, the position feature and the edge feature.
5. The teaching diagram visual positioning method based on adaptive Gestalt perception according to claim 3 is characterized in that: S320 includes: S321, in the current iteration, selecting high-level semantic features of a teaching schematic diagram, visual features of blocks in the teaching schematic diagram, and text description word features as input in the multimodal feature dataset; S322, for any two blocks in the input teaching schematic diagram, the adaptive Gestalt perception factors corresponding to the color features, position features and edge features of the arbitrary two blocks are compared with the probability threshold; if the adaptive Gestalt perception factor is greater than the probability threshold, the color features, position features and edge features of the arbitrary two blocks are correspondingly fused to obtain the adaptive Gestalt perception features of the input teaching schematic diagram.
6. The teaching diagram visual positioning method based on adaptive Gestalt perception according to claim 3 is characterized in that: S330 includes: S331, defining the number of attention heads and channel mapping dimensions of the multi-head attention mechanism in the visual text blender; S332, using the adaptive Gestalt perception features of the teaching schematic diagram as keys and values in a multi-head attention mechanism, and using the high-level semantic features of the teaching schematic diagram as a query; S333, calculating the attention scores of the adaptive Gestalt perception features and the high-level semantic features, and using the attention scores to screen the most effective adaptive Gestalt perception features for the high-level semantic representation, and obtaining the teaching schematic representation enhanced by adaptive Gestalt perception by weighted summation; S334, using the text description word element feature as a key and a value, and the high-level semantic feature of the teaching schematic diagram as a query, calculating the attention score of the text description word element feature and the high-level semantic feature of the teaching schematic diagram, using the attention to obtain the text description word element feature that is most effective in screening the high-level semantic representation, and obtaining the teaching schematic diagram representation enhanced by the text word element by weighted summation; S335, taking the teaching schematic representation enhanced by adaptive Gestalt perception and the teaching schematic representation enhanced by text words as input, calculating the channel attention and spatial attention of the features, and performing weighted summation to obtain the fusion features of the semantically enhanced teaching schematic.
7. The teaching diagram visual positioning method based on adaptive Gestalt perception according to claim 6 is characterized in that: S340 includes: S341, defining the number of attention heads and feature mapping dimensions of the multi-head attention mechanism in the visual localization decoder; S342, using the fused features of the semantically enhanced teaching schematic diagram as the key and value in the attention mechanism, calculating the attention score of the target query and the fused features, and screening the target features of the teaching schematic diagram that are most relevant to the text description; wherein the target query is a predefined learnable parameter; S343, decoding the most relevant target features of the target query into the spatial position coordinates of the object area corresponding to the description text on the teaching schematic diagram and marked with a prediction box.
8. The teaching diagram visual positioning method based on adaptive Gestalt perception according to claim 3 is characterized in that: S350 includes: The spatial position coordinates of the object area marked by the prediction box and the spatial position coordinates of the object area marked by the annotation box are input into the optimization loss function formula, and the loss value of the input teaching schematic diagram is calculated, which is expressed as: Among them, λ giou and represents the weight balancing factor; b represents the spatial position coordinates of the object area marked by the prediction box; Represents the spatial position coordinates of the object area marked by the annotation box; represents the intersection-over-union loss function, represents the mean absolute error loss function; Among them, A c is the minimum closure area of the prediction box and the annotation box, U is the union of the prediction box and the annotation box, and b i is the i-th prediction box in the teaching diagram, is the i-th annotation box of the teaching diagram, and N is the total number of teaching diagrams in the sample set.
9. The teaching diagram visual positioning method based on adaptive Gestalt perception according to claim 1 is characterized in that: Before S400, the teaching schematic diagram visual positioning method based on adaptive Gestalt perception also includes: Acquire a test teaching schematic diagram, and use a trained visual positioning model to perform visual positioning on the test teaching schematic diagram to obtain the spatial position coordinates of the object area corresponding to the description text of the test teaching schematic diagram; evaluating the performance of the visual localization model using the spatial position coordinates of the object area corresponding to the description text of the test teaching schematic diagram; The visual localization model whose performance meets the requirements is used as the screened visual localization model.
10. A teaching diagram visual positioning device based on adaptive Gestalt perception, characterized in that: include: An acquisition module is configured to acquire a sample set including a plurality of teaching schematic diagrams; An extraction module is configured to extract high-level semantic features, text description features and visual features of each teaching schematic diagram in the sample set, and construct a multimodal feature data set of the sample set using the high-level semantic features, visual features and text description features; A training module is configured to train a pre-built visual positioning model using the multimodal feature data set to obtain a trained visual positioning model; The positioning module is configured to screen the trained visual positioning model. After the screening is completed, it is deployed on the electronic device to perform visual positioning on the positioning teaching diagram and display it to the user.
