AR intelligent labeling method and system based on display content

Abstract

The application provides an AR intelligent labeling method and system based on display content, and relates to the technical field of intelligent labeling. The method comprises the following steps: acquiring a real-time image, performing instance segmentation to obtain a plurality of segmentation blocks and a semantic label of each segmentation block, integrating all the semantic labels into a semantic label library, and matching a target label from the semantic label library by using an attention-weighted similarity algorithm after acquiring a user field of view; acquiring data such as the minimum distance from an AR device to a display interface, calculating a target distance from the AR device to the centroid of each segmentation block to which a semantic label belongs by using the Pythagorean theorem, and superimposing and displaying the target label in the field of view by using an augmented reality engine based on the target distance. The application can consider the spatial positional relationship between an AR device and a display interface of a power equipment when labeling the power equipment, thereby reducing the problem of spatial misplacement of labeled information and actual equipment components.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent labeling, in particular to an AR intelligent labeling method and system based on display content. BACKGROUND

[0002] In the daily operation, maintenance and training scenes of the power industry, accurate identification and information labeling of power equipment are crucial. With the continuous advancement of smart grid construction, the types of power equipment are increasingly diverse and the structure is increasingly complex. The traditional method of relying on manual carrying of drawings and manuals for equipment identification and information query has been difficult to meet the efficient and accurate operation needs. The AR labeling technology based on display content has become a key direction to solve this problem because it can superimpose virtual information on real power equipment scenes in real time. Through this technology, workers can directly view device models, operating parameters, maintenance records and other key information in the real power equipment field of view with the help of AR devices, greatly improving equipment identification efficiency and operation safety. In particular, in the scenes of substation inspection, power equipment fault troubleshooting and new employee training, it has irreplaceable application value and promotes the transformation of the power industry to intelligent and digital operation mode.

[0003] Currently, in the field of display content AR labeling of power equipment identification, the mainstream scheme mainly falls into two categories. One is an AR labeling method based on preset template matching. This method needs to pre-construct a standard image template library of a large number of power equipment. In actual application, the real-time image obtained by the AR device is compared with the images in the template library to determine the matching device type, and then the preset labeling information is called to be superimposed and displayed. The other is a semantic recognition labeling technology based on a basic deep learning model. This technology uses CNN (Convolutional Neural Network) and other models to extract features and perform semantic segmentation on the real-time obtained power equipment images to obtain the segmentation areas corresponding to different equipment components. Then, the preset equipment information database is combined to match the text labeling information for the segmentation areas, and the AR engine is used to superimpose the labeling information in the real-time field of view.

[0004] The existing power equipment display content AR labeling technology does not consider the spatial position relationship between the AR device and the power equipment display interface. In the labeling information superimposition process, only the labeling text is simply associated with the segmentation area for display, which is prone to the problem of spatial misplacement of the labeling information and the actual equipment components, affecting the accurate reading of the labeling information by the workers and reducing the operation efficiency. SUMMARY

[0005] In order to consider the spatial position relationship between the AR device and the power equipment display interface when labeling the power equipment and reduce the problem of spatial misplacement of the labeling information and the actual equipment components, the present application provides an AR intelligent labeling method and system based on display content.

[0006] In a first aspect, the application provides an AR intelligent labeling method based on display content, which adopts the following technical solution:

[0007] An AR intelligent labeling method based on display content, comprising the following steps:

[0008] Obtaining a real-time image, performing instance segmentation processing on the real-time image to obtain a plurality of segmentation blocks and semantic labels of each segmentation block, integrating all the semantic labels into a semantic label library, obtaining a field of view of a user, obtaining semantic labels contained in the field of view, denoted as target labels;

[0009] Obtaining a minimum distance from an AR device to a display interface, a projection point of the AR device on the display interface, a distance from the projection point to a centroid of a segmentation block to which each semantic label belongs, calculating a distance from the AR device to the centroid of the segmentation block to which each semantic label belongs using the Pythagorean theorem, denoted as a target distance, and based on the target distance, superimposing and displaying the target labels in the field of view using an augmented reality engine.

[0010] The application can capture semantic information in the real-time image by obtaining the real-time image and performing instance segmentation on it, so that the generated semantic labels are more accurate and comprehensive. The application can closely combine with the user's visual perception by obtaining the field of view of the user and obtaining the corresponding semantic labels in the field of view to obtain the target labels, and then superimposing and displaying the target labels in the field of view, thereby enhancing the naturalness and intuitiveness of the interaction. The user does not need to actively search for information, and the application can automatically present the target labels according to the user's line of sight direction, thereby improving the user experience.

[0011] The application provides a spatial positioning basis for accurate display of the labels in the AR field of view by obtaining the minimum distance from the AR device to the display interface, the projection point, and the distance from the projection point to the centroid of the segmentation block to which each semantic label belongs, and calculating the target distance from the AR device to the centroid of the segmentation block to which each semantic label belongs using the Pythagorean theorem, so that the target labels can be superimposed in the real scene with appropriate proportions and positions, thereby enhancing the fusion degree of virtual information and the real environment and enabling the user to obtain a more realistic AR experience. The application uses the augmented reality engine to superimpose and display the target labels in the field of view based on the calculated target distance, so that the target labels can be naturally integrated into the user's visual environment without being conspicuous or inconsistent, thereby further improving the practicality and visual effect of the system.

[0012] Optionally, the method further comprises:

[0013] The gaze point heat map of the user is obtained, the average gaze time of each semantic label in the semantic label library is calculated according to the gaze point heat map, the sequence weight of each semantic label is calculated based on the average gaze time of each semantic label, the semantic labels are sorted in descending order of the sequence weight, and a hierarchical sequence is obtained.

[0014] The depth interval is calculated according to the depth information of the clipping space, and the target distance of each semantic label is adjusted according to the depth interval and the position of each semantic label in the hierarchical sequence.

[0015] According to the gaze point heat map of the user, the average gaze time of each semantic label in the semantic label library is calculated, and then the sequence weight of each semantic label is obtained. The sequence weight fully considers the actual attention degree of the user to different semantic labels. In the actual use scene, the user often concentrates more attention on the parts or regions that he considers important. By using the above scheme, the attention focus can be intuitively reflected, and the personalized presentation of the target label is realized.

[0016] The semantic labels are sorted in descending order of the sequence weight of each semantic label to obtain a hierarchical sequence. The hierarchical sequence establishes a priority system for the information that the user pays attention to, so that the information that the user pays most attention to can be timely and accurately presented, and the efficiency and pertinence of information transmission are improved.

[0017] The depth interval is calculated according to the depth information of the clipping space, which provides a reasonable space division basis for subsequent adjustment of the target distance of the semantic label. The depth information of the clipping space reflects the distance relationship between the objects in the scene and the observer. By calculating the depth interval, the entire scene space can be divided into different hierarchical regions. The target distance of each semantic label is adjusted according to the depth interval and the position of each semantic label in the hierarchical sequence, so as to combine the attention degree of the user to the semantic label with the position of the label in the space. The semantic label with a higher weight (i.e., the semantic label that the user pays more attention to) can be more prominent when displayed, thereby creating a clear hierarchical visual effect in the AR field of view, enhancing the spatial sense and reality sense of the fusion of virtual information and real scene, and enabling the user to more clearly distinguish information of different importance.

[0018] Optionally, the method further includes setting a maximum length of the hierarchical sequence, and if the number of semantic labels is greater than the maximum length of the hierarchical sequence, merging the semantic labels exceeding the maximum length of the hierarchical sequence.

[0019] The application effectively reduces the amount of information that needs to be processed and rendered by setting the maximum length of the hierarchical sequence and merging semantic labels that exceed this length. This reduces the computational burden and improves overall response speed and stability. The display of the merged semantic labels in the AR field of view is more concise and clear, reducing visual interference and confusion. Users can more easily identify and understand the information represented by the semantic labels, improving information acquisition efficiency. Limiting the length of the hierarchical sequence and merging semantic labels conforms to users' cognitive habits, allowing users to interact with the AR system more naturally. Users do not need to spend excessive effort to filter and understand a large amount of information, but can quickly grasp the key points, improving user satisfaction and comfort when using AR applications.

[0020] Optionally, the method further comprises:

[0021] The difference between the depth values of the pixel points contained in the i-th layer semantic label and the j-th layer semantic label in the hierarchical sequence is calculated. It is determined whether there is a difference in depth values less than a preset depth threshold. If so, the pixel point pair corresponding to the difference in depth values less than the preset depth threshold is recorded as a conflict pixel point pair, and each pixel point in the conflict pixel point pair is recorded as a conflict pixel point. If not, no processing is performed.

[0022] The spatial coordinates of the conflict pixel points in the frustum are calculated by inverse projection transformation. The mixing coefficient is calculated according to the relative positions of the conflict pixel points in the frustum. The pixel values of the conflict pixel points are corrected based on the mixing coefficient.

[0023] By calculating the difference between the depth values of the pixel points contained in the i-th layer and the j-th layer semantic labels in the hierarchical sequence and comparing it with the preset depth threshold, the application can accurately determine whether there is a depth conflict, i.e., whether there is a conflict pixel point. Because the depth value reflects the depth information of the pixel point in the scene, when the depth values of the pixel points of the two layers of semantic labels are less than the preset depth threshold, it means that they may be too close in vision, with the risk of overlap or conflict. After identifying the conflict pixel points, the application converts them into spatial coordinates in the frustum by inverse projection transformation, which can more intuitively understand the relative positional relationship of the conflict pixel points under the camera view. The mixing coefficient is calculated according to their relative positions in the frustum, and the pixel values of the conflict pixel points are corrected based on the coefficient, which can make the visual effect of the conflict area more natural and harmonious, improving the overall visual quality.

[0024] In an augmented reality scenario, virtual semantic labels need to be fused with the real scene. When semantic labels of different layers have depth conflicts, the fusion of the real scene is affected. The application solves the conflict problem through the above method, can make the virtual label better integrate into the real environment, enhance the acceptance and immersion of the user to the virtual information, and help to create a clearer spatial hierarchy. By correcting the pixel value of the conflict pixel point, the application can clearly distinguish the front and back position relationship of different semantic labels in space, so that the user can more accurately perceive the spatial distribution of virtual information.

[0025] Optionally, the method further comprises:

[0026] obtaining a main video stream and an auxiliary video stream, extracting target feature points in each frame image of the auxiliary video stream by using a feature point extraction algorithm, matching the target feature points with feature points in corresponding frame images of the main video stream by using a feature point matching algorithm, and obtaining a matching result;

[0027] According to the matching result, a transformation matrix of the main video stream and the auxiliary video stream is calculated, and a current frame image of the main video stream after geometric transformation is taken as a real-time image according to the transformation matrix.

[0028] The application can accurately find the corresponding relationship between the two video streams by extracting target feature points in the auxiliary video stream and matching them with the feature points in the corresponding frame images of the main video stream. Based on the matching result, the application calculates the transformation matrix and performs geometric transformation on the main video stream, which can accurately align the main video stream and the auxiliary video stream in space. The videos taken by different cameras may have different angles of view, and the above solution can fuse the video streams of different cameras to make the monitoring picture more comprehensive and coherent, which is convenient for comprehensive analysis and monitoring of the scene.

[0029] Accurate feature point matching and geometric transformation can avoid obvious misalignment and distortion in the video fusion process, making the fused video more natural and smooth in vision. By matching the feature points of the auxiliary video stream with the main video stream and calculating the transformation matrix to perform geometric transformation on the main video stream, the virtual information can be more accurately superimposed into the real scene. During video shooting, due to the shaking or movement of the camera, the video picture may appear shaking and unstable. The application can correct the main video stream by matching the feature points of the auxiliary video stream with the main video stream and performing geometric transformation, thereby reducing the influence of shaking.

[0030] Optionally, after obtaining the main video stream and the auxiliary video stream, the method further comprises:

[0031] The anti-shake algorithm is used to detect the shaking parameters in the auxiliary video stream in real time, and an anti-shake signal is generated according to the shaking parameters. The anti-shake signal is fed back to the acquisition device of the main video stream in real time, and the acquisition device of the main video stream adjusts the acquisition parameters according to the anti-shake signal.

[0032] The anti-shake algorithm can detect the shaking parameters in the auxiliary video stream in real time, which reflect the shaking situation of the device during video shooting, such as the amplitude, frequency and direction of shaking. According to these parameters, the anti-shake signal is generated and fed back to the acquisition device of the main video stream. The acquisition device can timely adjust the acquisition parameters, such as changing the lens angle, adjusting the sensor position, etc., so as to effectively reduce the picture shaking in the main video stream, making the shot video more stable and clear. Stable video picture helps to better capture and present the details in the picture. When the picture shaking is reduced, the resolution and clarity of the image can be fully utilized, so that the user can clearly see the details such as the device and its components in the video.

[0033] Optionally, the method further comprises:

[0034] Obtaining real-time interactive operation of the user on the target label, the real-time interactive operation including gesture trajectory, voice instruction and eye focus;

[0035] Extracting interactive features based on real-time interactive operation, including: analyzing the curvature features and motion vectors of the gesture trajectory, converting the voice instruction into semantic vectors through natural language processing technology, and recording the focus residence time and pupil change parameters through eye tracking technology;

[0036] Integrating all interactive features into an interactive feature matrix, performing feature fusion on the interactive feature matrix through a convolutional neural network to generate adjustment parameters of the target label, the adjustment parameters including transparency and scaling coefficient of the target label.

[0037] The present application integrates gesture trajectory, voice instruction and eye focus, and fully utilizes the advantages of different modalities. Gesture trajectory can intuitively express the user's intention to adjust the position and size of the target label; voice instruction can quickly issue operation commands without manual operation; eye focus can accurately locate the area of user's attention, and when the user gazes at a target label for a long time, corresponding adjustment can be made accordingly, improving the intuitiveness and convenience of interaction.

[0038] The present application obtains real-time interactive operation of the user on the target label and generates adjustment parameters, so that the user can see the changes of the target label during operation, improving the fluency and continuity of interaction, enhancing the interaction between the user and the system, and enabling the user to focus more on the operation task, thus improving the overall interactive experience.

[0039] Different users may have different needs for the display of the target label. Some users want the label to be more transparent so as not to affect the viewing of the background content, and some users want the label to be enlarged so as to read the information more clearly. The adjustment parameters generated by the present application include transparency and scaling factor, which can meet the diversified customization needs of users, so that each user can adjust the target label according to his own preferences and use scene, thereby improving the user's satisfaction and loyalty.

[0040] Optionally, the method further comprises: establishing an interaction knowledge graph, associating the target label with the historical interaction operation of the user according to the interaction knowledge graph, calculating the interaction weight of the target label according to the association result, and taking the product of the interaction weight and the sequence weight as a new sequence weight.

[0041] The present application can closely associate the target label with the historical interaction operation of the user by establishing an interaction knowledge graph. By analyzing these associated information, the interest degree and use habit of the user for different target labels can be deeply understood, and a more personalized interaction experience can be provided. Associating and mapping the target label with the historical interaction operation is helpful to understand the context information of the current interaction of the user. When the user interacts with a certain target label again, the present application can refer to the previous relevant interaction record to better predict the intention and demand of the user.

[0042] The present application can comprehensively consider the interaction frequency and importance (i.e. interaction weight) of the user for the target label and the position and order (i.e. sequence weight) of the target label in the overall sequence by taking the product of the interaction weight and the sequence weight as a new sequence weight. By adopting the above scheme, the present application can more reasonably allocate resources and preferentially process the target label with high user attention and importance in the sequence. The optimized sequence weight can guide the AR device to more accurately push the target label, reduce unnecessary recommendation and display, and reduce the computing burden and resource consumption of the AR device.

[0043] Optionally, the method further comprises:

[0044] Periodically detecting the timeliness of each semantic label in the semantic label library, the timeliness being determined by the difference between the generation time of each semantic label and the current time;

[0045] If the difference between the generation time of the semantic label and the current time exceeds a preset timeliness threshold, the semantic understanding processing is performed again on the segmentation block to which the semantic label whose difference exceeds the preset timeliness threshold belongs, and the processing result is updated to the semantic label library.

[0046] The tags in the semantic tag library are used to describe and classify various information. Over time, some device components may age and deteriorate, causing the original semantic tags to become inaccurate or irrelevant. The present application detects the timeliness of the semantic tags periodically. When the difference between the tag generation time and the current time exceeds the preset timeliness threshold, the semantic understanding process is performed again for the related segmentation block, the tag content is updated in a timely manner, and the information in the semantic tag library is always kept up-to-date.

[0047] Accurate semantic tags are the key to information retrieval. When the semantic tags can reflect the latest state of the information in a timely manner, the user can obtain more search results that meet the needs when searching for information. The present application detects and updates the semantic tags periodically, can perceive the changes in the environment in a timely manner, and make corresponding adjustments.

[0048] In a second aspect, the present application provides an AR intelligent labeling system based on display content, which adopts the following technical solution:

[0049] An AR intelligent labeling system based on display content, comprising a memory and a processor,

[0050] The memory stores a computer readable storage medium;

[0051] When the processor processes the computer program stored on the computer readable storage medium, the method as described in the first aspect can be implemented.

[0052] In summary, the present application includes at least one of the following beneficial technical effects:

[0053] 1. The present application obtains the minimum distance from the AR device to the display interface, the projection point, and the distance from the projection point to the centroid of each semantic tag belonging to the segmentation block, and uses the Pythagorean theorem to calculate the target distance from the AR device to the centroid of each semantic tag belonging to the segmentation block, providing spatial positioning basis for accurate display of the tags in the AR field, so that the target tags can be superimposed in the real scene with appropriate scale and position, enhancing the fusion degree of virtual information and real environment, and providing users with a more realistic AR experience. Based on the calculated target distance, the present application uses an augmented reality engine to superimpose and display the target tags in the field of view, so that the target tags can naturally blend into the user's visual environment and will not appear conspicuous or inconsistent, further improving the practicality and visual effect of the system.

[0054] 2. The present application sorts the semantic tags according to the order of the sequence weight of each semantic tag from large to small, obtains a hierarchical sequence, and establishes a priority system for the information that the user is interested in, so that the information that the user is most interested in can be presented in a timely and accurate manner, improving the efficiency and pertinence of information transmission. BRIEF DESCRIPTION OF DRAWINGS

[0055] Figure 1 This is a flowchart of Embodiment 1 of this application;

[0056] Figure 2 This is a flowchart of Embodiment 2 of this application;

[0057] Figure 3 This is a flowchart of Embodiment 3 of this application;

[0058] Figure 4 This is a flowchart of Embodiment 4 of this application. Detailed Implementation

[0059] The following combination Figures 1 to 4 This application will be described in further detail.

[0060] Example 1: This example discloses an AR intelligent annotation method based on displayed content, referring to... Figure 1 The method includes: S11 data acquisition and S12 label overlay. First, real-time images are acquired and segmented to obtain multiple segmentation blocks and semantic labels for each segmentation block. All semantic labels are integrated into a semantic label library. After acquiring the user's field of view, an attention-weighted similarity algorithm is used to match target labels from the semantic label library. Simultaneously, data such as the minimum distance from the AR device to the display interface are acquired. The Pythagorean theorem is used to calculate the target distance from the AR device to the centroid of the segmentation block to which each semantic label belongs. Finally, based on the target distance, the augmented reality engine overlays and displays the target labels in the field of view. The execution process of each step in this embodiment is as follows:

[0061] S11 data acquisition involves capturing real-time images through AR devices or camera modules, using the YOLOv8s model to perform instance segmentation on the real-time images, and outputting multiple segmentation blocks and their category IDs. The category ID of each segmentation block is the semantic label of that segmentation block. Finally, the semantic labels of all segmentation blocks are integrated into a semantic label library.

[0062] The AR device uses its built-in inertial measurement unit to obtain the user's gaze direction (such as the head-mounted display's pitch and yaw angles), projects the gaze direction onto the image plane, defines the rectangular area encompassed by the field of view, and calculates the coordinates of the center point of the minimum bounding rectangle of each segment based on the segmentation mask of that segment, i.e., the centroid coordinates of that segment. Based on the centroid coordinates of all segmented blocks and the field of view coordinate range, the segmented blocks contained within the field of view are obtained, i.e., the centroid coordinates of that segment are within the field of view coordinate range.

[0063] In other embodiments, the IoU between each segmented block and the field of view can be calculated separately. If the IoU is greater than 85%, it is determined that the segmented block is contained within the user's field of view, thereby obtaining all the segmented blocks within the field of view.

[0064] S12 superimposes the label, and obtains the minimum distance between the AR device and the display interface through the distance sensor built in the AR device. According to the relationship between the coordinate system of the AR device and the coordinate system of the display interface, the projection point coordinates of the AR device on the display interface are calculated through coordinate transformation. Based on the projection point coordinates and the centroid coordinates of the segmentation block, the Euclidean distance algorithm is used to calculate the distance from the projection point to the centroid of each semantic label corresponding segmentation block.

[0065] According to the minimum distance and the distance from the projection point to the centroid of each semantic label corresponding segmentation block, the Pythagorean theorem is used to calculate the distance from the AR device to the centroid of each semantic label corresponding segmentation block, which is recorded as the target distance. Based on the target distance, the target label is superimposed and displayed in the field of view using the augmented reality engine.

[0066] The power equipment usually has a complex structure and a variety of layouts. The above scheme can adapt to various complex power scenarios by comprehensively considering the spatial position relationship. Whether it is in an indoor substation or an outdoor power transmission line, the power equipment can be accurately labeled. The accurate labeling frame can provide more accurate and clear information for the operator, helping them to quickly identify and understand the state and parameters of the power equipment. In the power equipment inspection, maintenance and other work, the operator can quickly locate the problem components with the help of AR labeling, thereby improving the work efficiency and accuracy.

[0067] Embodiment 2: Refer to Figure 2 The difference between this embodiment and embodiment 1 is that the method further comprises:

[0068] S21 constructs a hierarchical sequence, uses eye tracking technology to monitor the user's gaze movement in real time, records the fixation points of the user in observing each position of the power equipment display content, and after a period of data collection and statistics, visualizes these fixation points on the image plane to form a fixation point heat map. Different color areas in the fixation point heat map represent the frequency and duration of the user's gaze. The deeper the color, the more times or longer the time the area is gazed at, reflecting that the user pays more attention to these areas.

[0069] The semantic label library contains semantic labels corresponding to each component of the power equipment. For each semantic label, the total gaze time of the user when observing the area corresponding to the label is counted, and then the average of these gaze times is calculated to obtain the average gaze time of each semantic label.

[0070] The sequence weight of each semantic label is calculated based on the average gaze time of each semantic label. The longer the average gaze time, the more attention the user pays to the component corresponding to the semantic label, and the greater the sequence weight. In this embodiment, the average gaze time is normalized, and the result of the normalization is the sequence weight. After calculating the sequence weights of all semantic labels, the semantic labels are sorted in descending order of sequence weight to obtain a hierarchical sequence.

[0071] S22 adjusts the target distance and crops the space to define a spatial region of the range of the scene visible to the AR device. According to the depth information of the cropped space, the cropped space is divided into a plurality of equally spaced depth segments, and the length of each depth segment is the depth interval. In this embodiment, the calculation model of the depth interval is as follows:

[0072] ;

[0073] wherein, is the depth interval; is the maximum depth of the cropped space; is the minimum depth of the cropped space; is the number of depth intervals.

[0074] According to the position of each semantic label in the hierarchical sequence and the calculated depth interval, the target distance of each semantic label is adjusted. The semantic labels at the front of the hierarchical sequence are assigned smaller target distances, so that they are displayed closer and clearer in the AR field of view, while the semantic labels at the back are assigned larger target distances, so that they are displayed relatively farther and smaller. In this embodiment, the process of adjusting the target distance of each semantic label is as follows:

[0075] First, the minimum distance from the AR device to the display interface is adjusted according to the depth interval. The calculation model for adjustment is as follows:

[0076] ;

[0077] wherein, is the adjusted minimum distance, i.e., the minimum distance from the AR device to the depth segment where the kth semantic label in the hierarchical sequence is located.

[0078] Then, the segmentation block centroid coordinates are adjusted according to the depth segment to which each semantic label belongs. Taking the AR device as the coordinate origin, for example, the direction of the AR device towards the display interface is the positive direction of the vertical axis, the direction of the AR device towards the ground is the positive direction of the horizontal axis, and the right-hand rule determines the positive direction of the vertical axis. The vertical axis coordinate value of the segmentation block centroid coordinates is adjusted to the minimum distance .

[0079] Finally, according to the adjusted minimum distance and the adjusted segmentation block centroid coordinates, the target distance of each semantic label is adjusted using the Pythagorean theorem.

[0080] In other embodiments, in order to avoid the rendering efficiency being low or the display being chaotic due to the layered sequence being too long, a maximum length of the layered sequence is also set, and when the number of semantic labels is greater than the maximum length of the layered sequence, the semantic labels exceeding the maximum length are merged, and the merged semantic labels are updated to the non-interactive layer.

[0081] S23 conflict judgment, for the i-th layer and the j-th layer semantic labels in the layered sequence (i≠j), the depth values of the pixel points contained by them are calculated respectively, and the difference values of the depth values of any two pixel points at different levels are calculated according to the depth values of the pixel points contained by the i-th layer and the j-th layer semantic labels, and it is judged whether there is a case that the difference value of the depth values is less than a preset depth threshold. If there is, it means that the pixel points corresponding to the two layers of semantic labels exist conflict in depth, that is, they may be mutually occluded or not clearly displayed in the AR field of view, and the conflict pixel point pairs are marked, and each pixel point in the conflict pixel point pair is recorded as a conflict pixel point; if not, no processing is performed.

[0082] S24 processing conflict, the conflict pixel points are converted from the screen coordinate system to the view volume space coordinate system through inverse projection transformation, and the process is as follows:

[0083] The intrinsic matrix and the projection transformation matrix of the camera are obtained, and the 2D image coordinates of the conflict pixel points are combined with the depth values to convert them into 3D view volume space coordinates by using the inverse projection formula and , wherein, is the 3D coordinates of the conflict pixel points contained by the i-th layer semantic label, is the 3D coordinates of the conflict pixel points contained by the j-th layer semantic label.

[0084] The mixing coefficient is calculated according to the relative position of the conflict pixel point pair in the view volume space, and the mixing coefficient is used to determine how to fuse the pixel values of the two conflict pixel points when processing the conflict. The essence is the contribution ratio of the two conflict pixel points, which needs to be calculated according to the relative position of the two conflict pixel points in the 3D space, so that the fusion result meets the spatial perception of the human eye. The calculation model of the mixing coefficient is as follows:

[0085] ;

[0086] ;

[0087] ;

[0088] wherein d is the straight line distance of the two conflict pixel points in the view volume space. the mixing coefficient of the conflict pixel point contained in the i-th layer semantic label; the mixing coefficient of the conflict pixel point contained in the j-th layer semantic label.

[0089] In other embodiments, the mixing coefficient can also adopt the following calculation model:

[0090] ;

[0091] ;

[0092] wherein, the maximum depth of the clipping space; the minimum depth of the clipping space; the depth value of the conflict pixel point belonging to the i-th layer semantic label; the depth value of the conflict pixel point belonging to the j-th layer semantic label; the mixing coefficient of the conflict pixel point contained in the i-th layer semantic label; the mixing coefficient of the conflict pixel point contained in the j-th layer semantic label.

[0093] Based on the calculated mixing coefficient, the pixel values of the two conflict pixel points are weighted and averaged according to the mixing coefficient in a linear mixing manner to obtain a new pixel value. The calculation model of the new pixel value is as follows:

[0094] ;

[0095] ;

[0096] ;

[0097] wherein, the red channel value of the new pixel value; the red channel value of the conflict pixel point contained in the i-th layer semantic label; the red channel value of the conflict pixel point contained in the j-th layer semantic label; the green channel value of the new pixel value; the green channel value of the conflict pixel point contained in the i-th layer semantic label; the green channel value of the conflict pixel point contained in the j-th layer semantic label; the blue channel value of the new pixel value; the blue channel value of the conflict pixel point contained in the i-th layer semantic label; and the blue channel value of the conflict pixel point contained in the j-th layer semantic label.

[0098] By the above manner, the application can reduce the conflict between the pixels, and make the image display in the AR labeling scene clearer and more natural.

[0099] Embodiment 3: Refer to Figure 3 The difference between this embodiment and embodiment 1 is that the method further comprises:

[0100] S31 adjusts the acquisition parameters to obtain the main video stream and the auxiliary video stream, the main video stream and the auxiliary video stream are fixed on the same rigid support or carrier, the lens direction needs to be completely consistent or extremely close, so that the picture content shot by the two is greater than 80% overlap. The main video stream refers to a high-resolution or high-frame-rate stream, which is used for final output and maintains the original picture quality. The auxiliary video stream refers to a low-resolution or wide-angle stream (such as a super wide-angle lens), which is used for motion estimation.

[0101] The anti-shake algorithm is used to detect the jitter parameters (such as displacement and rotation angle) in the auxiliary video stream in real time, and based on the monitored jitter parameters, an anti-shake signal (i.e. compensation signal) is generated to offset the jitter, and the compensation signal is fed back to the acquisition device of the main video stream in real time. The acquisition device of the main video stream adjusts the acquisition parameters according to the compensation signal.

[0102] S32 feature point matching, for each frame of the auxiliary video stream, a feature point extraction algorithm (such as SIFT, ORB) is used to extract target feature points (such as edges and corner points) with uniqueness, the target feature points of the auxiliary stream are matched with the feature points of the corresponding frame of the main video stream, and the matching result is obtained by finding the corresponding pixel points in the two frames of images through the feature point matching algorithm.

[0103] S33 geometric transformation, according to the matching result, the transformation matrix (including translation, rotation, scaling and other parameters) of the main video stream and the auxiliary video stream is calculated, and the main video stream is geometrically transformed according to the transformation matrix, and the current frame image of the geometrically transformed main video stream is taken as the real-time image.

[0104] This embodiment can not only prevent shaking from the source through hardware, but also can be corrected again through software, so that the anti-shaking effect is more complete.

[0105] Embodiment 4: Refer to Figure 4 The difference between this embodiment and embodiment 2 is that the method further comprises:

[0106] S41 adjusts the target label to obtain the real-time interactive operation of the user on the target label, the real-time interactive operation includes gesture trajectory, voice instruction and eye movement focus.

[0107] The hand motion is captured by visual sensors (such as RGB cameras, depth cameras) or wearable sensors (such as inertial measurement units), the hand key points are located by a target detection model, and the motion parameters of the trajectory are calculated by time sequence tracking, including direction, speed, acceleration, and curvature change.

[0108] After the microphone array captures the voice signal, the effective audio is extracted after noise reduction processing, the voice recognition is converted into text, the text is matched with the voice instructions in the preset voice library, and the current voice instruction of the user is obtained.

[0109] The pupil center position and corneal reflection point are captured by an infrared camera and corneal reflection algorithm, the landing coordinates of the visual line on the display screen are calculated, the target label exceeding the preset gaze duration threshold is determined as focusing on the label, and the rapid eye movement from a target label to another target label indicates switching the focus target.

[0110] Based on real-time interactive operation, interactive features are extracted, including analyzing the curvature features and motion vectors of the gesture trajectory, converting the voice instruction into a semantic vector through natural language processing technology, and recording the focus dwell time and pupil change parameters through eye tracking technology, the process is as follows:

[0111] The curvature features and motion vectors of the trajectory are analyzed, the curvature features include sudden stop and circular degree, reflecting the operation intention intensity, and the motion vectors include x / y axis displacement direction and speed, corresponding to scaling or translation requirements.

[0112] The voice instruction is converted into a semantic vector through natural language processing technology.

[0113] The focus dwell time (the longer the dwell time, the higher the attention) and pupil change parameters (such as pupil diameter increase, corresponding to interest degree improvement) are extracted.

[0114] First, the above multi-dimensional interactive features are integrated into a unified interactive feature matrix, and the matrix is feature fused through a convolutional neural network (such as extracting cross-modal correlation features through a convolutional layer, and compressing redundant information through a pooling layer), and finally the adjustment parameters of the target label are output:

[0115] When the attention is high, the transparency is reduced, and when the attention is low, the transparency is increased;

[0116] When the voice instruction is zooming out or the gesture is expanding outward, the scaling coefficient is increased.

[0117] S42 adjusts the sequence weight, establishes an interactive knowledge graph, and the core of the knowledge graph is to establish an association network of semantic labels and user interaction behaviors:

[0118] The nodes of the knowledge graph include semantic labels, user interaction types (such as "click" and "voice query"), and interaction time.

[0119] The association between the edge representation label of the knowledge graph and the interaction feature (for example, the user has clicked a semantic label 3 times, and the user's voice query frequency for a semantic label is 2 times a day).

[0120] According to the association between the semantic label and the historical interaction operation of the user in the interaction knowledge graph, the interaction weight is calculated based on the association strength between the label and the historical interaction operation in the knowledge graph, and the product of the interaction weight and the sequence weight is taken as the new sequence weight.

[0121] S43 detects the timeliness, and periodically detects the timeliness of each semantic label in the semantic label library, which is determined by the difference between the generation time of each semantic label and the current time.

[0122] If the time difference of a certain label exceeds the preset timeliness threshold, the semantic understanding processing is performed again on the segmentation block to which the semantic label belongs, and the data collection of S11 is performed again on the image segmentation block to which the label belongs. If the re-identification result is consistent with the original label, the generation time of the semantic label is updated; if not, the identification result is used to replace the original semantic label and is updated to the library.

[0123] The present embodiment will be further described in combination with a specific case.

[0124] The inspection personnel are equipped with two kinds of interactive terminals, touch display screen and AR glasses, and the state label (including temperature, load rate, and running time) of each power equipment (such as No. 1 main transformer and No. 3 high-voltage circuit breaker) is displayed on the screen in real time. The inspection personnel can dynamically adjust the label display effect (transparency and size) through three interactive ways of gesture, voice, and eye movement.

[0125] S41 adjusts the target label, when the inspection personnel need to adjust the semantic label of No. 1 main transformer, the interactive operation is captured through the AR device and the internal configured sensor, and the key features are extracted, the hand movement is captured through the depth camera built-in the AR glasses, and the key points of the hand are positioned through the inertial measurement unit.

[0126] When the inspection personnel spread their hands outward to enlarge the label, the key points of the finger joints are positioned through the target detection model, and the motion parameters are calculated through the time sequence tracking: the direction is outward expansion, the speed is 0.5 m / s, the acceleration is 0.1 m / s², and the curvature change is close to 0.

[0127] The trajectory curvature feature (no sudden stop, low circularity, and clear operation intention) and the motion vector (x-axis positive displacement corresponding to the enlargement demand) are analyzed.

[0128] The patrol personnel say that the semantic label transparency of the No. 1 main transformer is adjusted to be low, the microphone array first reduces the environmental noise, and then converts the audio to text through voice recognition, matches with the preset voice library (including "enlarge the label", "lower the transparency", "switch to the circuit breaker label" and other instructions), and confirms that the current instruction is "adjust the semantic label transparency of the No. 1 main transformer".

[0129] Through natural language processing technology, the voice instruction is converted into a semantic vector ("No. 1 main transformer", "transparency", "lower").

[0130] The patrol personnel fix their gaze on the semantic label of "No. 1 main transformer", and the infrared camera combines the corneal reflection algorithm to capture the pupil center position and the corneal reflection point, and calculate the landing coordinates of the visual line on the screen; when the gaze duration exceeds 3 seconds, it is determined that "the label is focused on"; if the visual line is quickly moved to the semantic label of "No. 3 circuit breaker" subsequently, it is determined that "the focus target is switched".

[0131] The focus and stay duration (3.5 seconds), pupil change parameters (pupil diameter increases from 3 mm to 3.8 mm, indicating that the interest in the device is improved).

[0132] The trajectory curvature feature, motion vector, semantic vector, stay duration, and pupil parameter are integrated into an interaction feature matrix, and then each feature is converted into a fixed length vector. In this embodiment, each feature is converted into a 1x10 vector, 1 representing the number of samples and 10 representing the feature dimension.

[0133] The trajectory curvature feature uses a 10-dimensional vector to describe the number of sudden stops (the number of times the speed drops by more than 50% within 1 second), the average circular degree (the degree of curvature of the trajectory, a straight line is 0 and a circle is 1), the peak value of the curvature change rate (the maximum change amount of the circular degree per unit time), the high curvature interval ratio (the proportion of the trajectory length with a circular degree greater than 0.6 in the total length), the average stay duration after sudden stop (the duration of the speed less than 0.1 m / s after sudden stop), the trajectory smoothness (the variance of the curvature of adjacent time points, the smaller the value, the smoother), the maximum curvature position (the proportion of time at which the high curvature point appears, such as 0.2 representing 20% of the time), the curvature symmetry (the similarity of the curvatures of the left and right halves of the trajectory, 1 for complete symmetry), the deceleration stage curvature change (the circular degree increment when the speed decreases), and the acceleration stage curvature change (the circular degree increment when the speed increases).

[0134] In this embodiment, the vector value is [0, 0.2, 0.1, 0.3, 0.05, 0.1, 0.02, 0.08, 0.01, 0.03], wherein 0 represents no emergency stop, 0.2 represents low circularity, 0.1 represents that the curvature change is less than 1, 0.3 represents that the curvature of 30% of the trajectory is greater than 0.5, 0.05 represents that the duration of the speed less than 0.1 m / s after the emergency stop is 0.05, 0.1 represents high smoothness, the variance of the curvatures of adjacent time points is less than 0.5, 0.02 represents the time of the high curvature point, indicating 2% of the time, 0.08 represents that the similarity of the curvatures of the left and right half trajectories is less than 0.2, 0.01 represents that the curvature is almost unchanged when decelerating, and 0.03 represents that the curvature slightly increases when accelerating.

[0135] The motion vector is described by a 10-dimensional vector, including x-axis instantaneous speed (positive to the right, unit: m / s), y-axis instantaneous speed (positive upward, unit: m / s), x-axis cumulative displacement (relative to the starting point, unit: pixels), y-axis cumulative displacement (relative to the starting point, unit: pixels), x-axis acceleration (speed change rate, positive for acceleration, unit: m / s²), y-axis acceleration (speed change rate, unit: m / s²), motion direction angle (included angle with the positive direction of the x-axis, 0°-360°, normalized to 0-1), speed direction consistency (coincidence degree of motion directions of adjacent time points, 1 for complete coincidence), displacement ratio (x-axis displacement to total displacement), and speed peak time (time proportion of the appearance of the maximum speed).

[0136] In this embodiment, the vector value is [0.5, 0, 10, 0, 0.1, -0.02, 0.8, 0.05, 0.2, 0.01], wherein 0.5 represents an x-axis speed of 0.5 m / s, 0 represents no motion in the y-axis direction, 10 represents moving 10 pixels to the right from the starting point, 0 represents no displacement in the y-axis direction, 0.1 represents acceleration motion in the x-axis direction, -0.02 represents slight deceleration in the y-axis direction (in the power equipment interaction scene, the duration of a single action of a gesture is usually very short (such as 0.5-1 second), and if the y-axis speed itself is very small, even if there is a short deceleration, the cumulative displacement will tend to 0), 0.8 is defined according to the normalized motion direction angle, representing the normalized value of the included angle with the positive direction of the x-axis, 0.05 is defined according to the speed direction consistency, representing that the coincidence degree of motion directions of adjacent time points is 0.05, 0.2 represents that the x-axis displacement ratio is 20%, and 0.01 represents that the speed peak time appears at 1% of the time.

[0137] The semantic vector is described by a 10-dimensional vector, including operation type: adjusting low parameter, device object: No. 1 main transformer, device object: No. 3 circuit breaker, parameter type: transparency, parameter type: scaling coefficient, operation type: adjusting high parameter, operation type: translating label, device object: No. 2 isolating switch, parameter type: display position, and emergency degree (1 for the highest).

[0138] In the embodiment, the vector value of the voice instruction "lower the transparency of the No. 1 main transformer" is [0.9, 0.1, 0, 0.8, 0, 0, 0, 0, 0, 0.2], wherein 0.9 represents that the weight of the lower parameter is the highest, 0.1 represents that the adjustment object this time is the No. 1 main transformer, 0 represents that it is irrelevant to the No. 3 circuit breaker, 0.8 represents that the adjustment this time is the transparency, 0 represents that it is irrelevant to the scaling coefficient, 0 represents that it is irrelevant to the higher parameter, 0 represents that it is irrelevant to the translation label, 0 represents that it is irrelevant to the No. 2 disconnector, 0 represents that it is irrelevant to the display position, and 0.2 represents low emergency degree. In the embodiment, [0.8, 1] represents high emergency degree, (0.8, 0.4] represents medium emergency degree, and (0.4, 0] represents low emergency degree.

[0139] The dwell time is converted into a focus degree interval coding value by using a 10-dimensional vector. The continuous dwell time is mapped to 10 dimensions through time interval discretization, each dimension corresponds to a preset interval, and the value of 0-1 represents the probability that the time length belongs to the interval (only one dimension is 1 and the rest are 0, that is, a variant of one-hot coding). The 10-dimensional vector of the dwell time of 3.5 seconds is shown in Table 1.

[0140] Table 1

[0141] Time interval 3.5 seconds encoding value Meaning [0,0.5) 0 3.5 seconds not belonging to the interval, weight 0 [0.5,1) 0 3.5 seconds not belonging to the interval, weight 0 [1,2) 0 3.5 seconds not belonging to the interval, weight 0 [2,3) 0 3.5 seconds not belonging to the interval, weight 0 [3,4) 1 3.5 seconds belonging to the interval, weight 1 [4,5) 0 3.5 seconds not belonging to the interval, weight 0 [5,6) 0 3.5 seconds not belonging to the interval, weight 0 [6,7) 0 3.5 seconds not belonging to the interval, weight 0 [7,8) 0 3.5 seconds not belonging to the interval, weight 0 ≥8 0 3.5 seconds not belonging to the interval, weight 0

[0142] The pupil parameter uses a 10-dimensional vector to describe the pupil diameter mean (unit: mm), diameter change amount (end-start, unit: mm), diameter change rate (change amount divided by total time, unit: mm / s), contraction times (number of times the diameter decreases by more than 0.2 mm within 1 second), expansion times (number of times the diameter increases by more than 0.2 mm within 1 second), maximum diameter occurrence time (proportion relative to total time, 0%-100%), diameter fluctuation amplitude (standard deviation, unit: mm), blink interference coefficient (proportion of diameter anomalies caused by blinking), interest degree prediction value (normalized score based on diameter change), data confidence (sensor recognition clarity, 1 is the highest).

[0143] In the embodiment, the vector corresponding to the pupil parameter is [3.8, 0.8, 0.2, 0.1, 0.05, 0.3, 0.1, 0.02, 0.3, 0.01], wherein 3.8 represents the pupil diameter mean, 0.8 represents the diameter change amount of 0.8, indicating that the pupil is significantly enlarged, 0.2 represents the diameter change rate of 0.2, indicating that the enlargement speed is relatively fast, 0.1 represents almost no contraction, 0.05 represents single slight expansion, 0.3 represents that the maximum diameter occurs at the 30% time length, 0.1 represents small fluctuation, 0.02 represents almost no blink interference, 0.3 represents medium to high interest degree, and 0.01 represents low confidence.

[0144] Stack 5 1x10 eigenvectors by row to form a 5x10 two-dimensional matrix (rows represent feature types, columns represent feature dimensions), namely the interaction feature matrix.

[0145] The convolution layer of the convolutional neural network slides over the interaction feature matrix through a 3x3 convolution kernel, calculates the sum of the product of values in the local area each time, and generates a new feature map, thereby capturing the correlation between different features.

[0146] The pooling layer of the convolutional neural network slides over the feature map output by the convolution layer by setting a 2x2 pooling window, and each time it slides, it takes the maximum value of the four values in the pooling window as a value in the new feature map, thereby compressing redundant data.

[0147] After correlation extraction by the convolution layer and redundancy compression by the pooling layer, the convolutional neural network inputs the final feature map into the fully connected layer to convert it into specific label adjustment parameters, such as:

[0148] Based on the strong correlation between speech and eye movement, it is determined that the user's semantic label attention to No. 1 main transformer is high, and the transparency is reduced from 60% to 30%, and combined with the motion vector (expanding outward) feature captured by the convolution layer, the label size is enlarged by 1.5 times.

[0149] The final output adjustment parameters are: the semantic label transparency of No. 1 main transformer is reduced from 60% to 30% (high attention reduces transparency), and the label size is enlarged by 1.5 times (gesture expansion corresponds to scaling needs).

[0150] S42 adjusts the sequence weight, establishes an electric power equipment interaction knowledge graph, combines historical operation adjustment label sequence weight, and displays high-frequency attention labels first.

[0151] The nodes of the knowledge graph include: semantic labels (No. 1 main transformer, No. 3 circuit breaker, No. 2 disconnecting switch), user interaction types (gesture enlargement, voice query, eye movement attention), and interaction time (2024-05-01 09:15, 2024-05-01 09:20).

[0152] The edges of the knowledge graph include: the inspection personnel has enlarged the No. 1 main transformer semantic label 5 times through gesture; the inspection personnel queries the No. 3 circuit breaker load rate through voice 2 times a day; and the average eye movement attention time of the inspection personnel to the No. 2 disconnecting switch label is only 1 second.

[0153] Based on the correlation strength of semantic labels and historical interactions in the knowledge graph, calculate the interaction weight: No. 1 main transformer interaction weight 0.8, No. 3 circuit breaker interaction weight 0.6, and No. 2 disconnecting switch interaction weight 0.2.

[0154] Assuming that the initial sequence weights are all 1.0, the new sequence weight = interaction weight x initial sequence weight, the new weight of No. 1 main transformer is 0.8, the new weight of No. 3 circuit breaker is 0.6, and the new weight of No. 2 disconnector is 0.2.

[0155] When the subsequent system displays the semantic labels, the semantic label of the No. 1 main transformer with a high weight is preferentially placed in a conspicuous position on the screen.

[0156] S43 detects the time limit, and periodically detects the timeliness of the semantic labels of all power equipment, so as to reduce the misjudgment of inspection caused by outdated labels.

[0157] The preset timeliness threshold is 7 days, that is, when the difference between the label generation time and the current time exceeds 7 days, the resemantic understanding process needs to be triggered, that is, the data collection of S11 is re-executed.

[0158] Taking the semantic label of No. 2 disconnector as an example, the generation time is 2024-04-20, the current time is 2024-05-01, and the difference is 11 days (exceeding the threshold of 7 days).

[0159] The data collection of S11 is re-executed for the image segmentation block where the No. 2 disconnector is located.

[0160] If the re-identification result is No. 2 disconnector, which is consistent with the original semantic label, only the label generation time is updated to 2024-05-01.

[0161] If the re-identification result is No. 1 main transformer, the original semantic label is replaced by No. 1 main transformer, and is updated to the semantic label library.

[0162] Embodiment 5: The embodiment discloses an AR intelligent labeling system based on display content, which comprises a memory and a processor,

[0163] The computer readable storage medium is stored in the memory;

[0164] When the processor processes the computer program stored on the computer readable storage medium, the AR intelligent labeling method based on display content can be realized.

[0165] The above are preferred embodiments of the present application, and do not limit the protection scope of the present application, so that: any equivalent changes made on the basis of the structure, shape, principle of the present application should be covered within the protection scope of the present application.

Claims

1. An AR intelligent annotation method based on displayed content, characterized in that, include: Acquire real-time images, perform instance segmentation on the real-time images to obtain multiple segmentation blocks and semantic labels for each segmentation block, integrate all semantic labels into a semantic label library, acquire the user's field of view, acquire the semantic labels contained within the field of view, and record them as target labels. The minimum distance from the AR device to the display interface, the projection point of the AR device on the display interface, and the distance from the projection point to the centroid of the segmentation block to which each semantic label belongs are obtained. The distance from the AR device to the centroid of the segmentation block to which each semantic label belongs is calculated using the Pythagorean theorem and recorded as the target distance. Based on the target distance, the target label is superimposed and displayed in the field of view using the augmented reality engine. The method further includes: Obtain the user's gaze heatmap, calculate the average gaze time of each semantic tag in the semantic tag library based on the gaze heatmap, calculate the sequence weight of each semantic tag based on the average gaze time of each semantic tag, and sort the semantic tags in descending order of sequence weight to obtain a hierarchical sequence. The depth interval is calculated based on the depth information of the clipping space, and the target distance of each semantic tag is adjusted based on the depth interval and the position of each semantic tag in the hierarchical sequence.

2. The AR intelligent annotation method based on displayed content according to claim 1, characterized in that, The method further includes: setting a maximum length for the hierarchical sequence; if the number of semantic tags is greater than the maximum length of the hierarchical sequence, then merging semantic tags that exceed the maximum length of the hierarchical sequence.

3. The AR intelligent annotation method based on displayed content according to claim 2, characterized in that, The method further includes: Calculate the difference between the depth values ​​of the pixels contained in the semantic labels of the i-th layer and the depth values ​​of the pixels contained in the semantic labels of the j-th layer in the hierarchical sequence. Determine whether there is a difference in depth values ​​less than a preset depth threshold. If so, record the pixel pair corresponding to the difference in depth values ​​less than the preset depth threshold as a conflicting pixel pair, and record each pixel in the conflicting pixel pair as a conflicting pixel. If not, do not process it. The view frustum coordinates of conflicting pixels are calculated by inverse projection transformation. The mixing coefficient is calculated based on the relative position of the conflicting pixels in the view frustum space. The pixel value of the conflicting pixels is then corrected based on the mixing coefficient.

4. The AR intelligent annotation method based on displayed content according to any one of claims 1-3, characterized in that, The method further includes: The main video stream and the auxiliary video stream are acquired. The target feature points are extracted in each frame of the auxiliary video stream using a feature point extraction algorithm. The target feature points are then matched with the feature points in the corresponding frame of the main video stream using a feature point matching algorithm to obtain the matching results. Based on the matching results, the transformation matrices of the main video stream and the auxiliary video stream are calculated. The main video stream is then subjected to a geometric transformation based on the transformation matrices, and the current frame image of the geometrically transformed main video stream is used as the real-time image.

5. The AR intelligent annotation method based on displayed content according to claim 4, characterized in that, After acquiring the main video stream and the secondary video stream, the method further includes: The anti-shake algorithm is used to detect jitter parameters in the auxiliary video stream in real time, generate an anti-shake signal based on the jitter parameters, and feed the anti-shake signal back to the acquisition device of the main video stream in real time. The acquisition device of the main video stream adjusts the acquisition parameters according to the anti-shake signal.

6. The AR intelligent annotation method based on displayed content according to any one of claims 1-3, characterized in that, The method further includes: The system acquires real-time interactive operations of the user on the target tag, including gesture trajectories, voice commands, and eye-tracking focus. Based on real-time interactive operations, interactive features are extracted, including: parsing the curvature features and motion vectors of gesture trajectories, converting voice commands into semantic vectors through natural language processing technology, and recording focus dwell time and pupil change parameters through eye-tracking technology. All interactive features are integrated into an interactive feature matrix. The interactive feature matrix is ​​then fused using a convolutional neural network to generate adjustment parameters for the target label. These adjustment parameters include the opacity and scaling factor of the target label.

7. The AR intelligent annotation method based on displayed content according to claim 6, characterized in that, The method further includes: establishing an interaction knowledge graph, associating semantic tags with the user's historical interaction operations based on the interaction knowledge graph, calculating the interaction weight of the semantic tags based on the association results, and using the product of the interaction weight and the sequence weight as the new sequence weight.

8. The AR intelligent annotation method based on displayed content according to claim 7, characterized in that, The method further includes: The timeliness of each semantic tag in the semantic tag library is periodically checked, and the timeliness is determined by the difference between the generation time of each semantic tag and the current time. If the difference between the semantic tag generation time and the current time exceeds a preset timeliness threshold, the segmentation block to which the semantic tag belongs will be re-semantically understood, and the processing result will be updated in the semantic tag library.

9. An AR intelligent annotation system based on displayed content, characterized in that, include: Memory and processor The memory contains a computer-readable storage medium; When the processor processes the computer program stored on the computer-readable storage medium, it is able to implement the method as described in any one of claims 1-8.

Images