A data recognition system for audio-visual glasses

Through the data identification system of audio-visual glasses, the eyeball direction and motion information are used to independently screen and enlarge valuable information, which solves the problem that audio-visual glasses cannot effectively screen information, improves the interactive experience and stabilizes the information display.

CN118710891BActive Publication Date: 2024-11-15SHENZHEN TIANQU XINGKONG TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411201124.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-29
Publication Date
2024-11-15
Estimated Expiration
2044-08-29

AI Technical Summary

Technical Problem

At this stage, audio-visual glasses cannot independently filter out information that is valuable to the wearer and cannot improve the interactive experience.

Method used

A data recognition system for audio-visual glasses is designed, including external vision module, acceleration module, analysis unit, display module, internal vision module, network module and sound module. By analyzing the wearer's eyeball direction and the movement information of audio-visual glasses, The main items that the wearer is concerned about are independently selected, and the inquired information is enlarged and offset to improve the stability of information display.

Benefits of technology

It realizes independent screening of valuable information, improves the interactive experience, and avoids screen shaking through offset correction, reducing the wearer's dizziness.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118710891B_ABST
    Figure CN118710891B_ABST
Patent Text Reader

Abstract

The present invention discloses a data recognition system for audio-visual glasses, and relates to the technical field of audio-visual glasses. The present invention comprises an external visual module, an acceleration module, an analysis unit, a display module, an internal visual module, a network module and a sound module, wherein the external visual module obtains image information w in front of the audio-visual glasses and transmits it to the analysis unit, the internal visual module obtains image information n of a single eyeball of the wearer and transmits it to the analysis unit, the acceleration module obtains spatial acceleration j of the audio-visual glasses and transmits it to the analysis unit, the analysis unit executes an eyeball tracking program to identify the eyeball direction of the wearer in the image information n, and the analysis unit executes a spatial positioning program to obtain the offset of the audio-visual glasses. The present invention can autonomously filter valuable information from the picture of the lens by identifying the eyeball direction of the wearer, and perform offset correction on the framed items in the picture of the lens by calculating the offset of the audio-visual glasses.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of audio-visual glasses, and in particular to a data recognition system for audio-visual glasses. Background Art

[0002] Audiovisual glasses are a type of smart glasses that can provide users with both viewing and listening experiences. Audiovisual glasses usually use a micro display or optical solution to reflect images onto the transparent lenses of the glasses, achieving a viewing effect similar to a giant screen. Audiovisual glasses have a high resolution and can provide clear and vivid images. Audiovisual glasses are equipped with speakers on both sides of the temples, so users can enjoy audio content while wearing the glasses.

[0003] At this stage, audio-visual glasses only have simple playback and display functions. The information queried through the Internet has no priority, and cannot effectively filter out information that is valuable to the wearer, and cannot improve the interactive experience. Therefore, how audio-visual glasses can autonomously filter valuable information is a technical problem that technical personnel in this field need to solve. Summary of the invention

[0004] In view of the deficiencies of the prior art, the present invention provides a data recognition system for audio-visual glasses, which solves the problems raised in the above-mentioned background technology.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solutions: a data recognition system for audio-visual glasses, including an external visual module, an acceleration module, an analysis unit, a display module, an internal visual module, a network module and a sound module, wherein the external visual module obtains image information w in front of the audio-visual glasses and transmits it to the analysis unit, the internal visual module obtains image information n of a single eyeball of the wearer and transmits it to the analysis unit, the acceleration module obtains spatial acceleration j of the audio-visual glasses and transmits it to the analysis unit, the external visual module and the internal visual module are both existing image sensors, and any image sensor on the market that can meet the requirements of high-definition image acquisition can be used, the display module is a screen display component of existing AR glasses, and any screen display component on the market that can meet the display function can be used, the network module is a wireless network card with wireless communication function, and any wireless network card on the market that can meet the requirements of wireless network connection can be used, the acceleration module is an existing acceleration sensor, and any acceleration sensor on the market that can meet the requirements of acceleration acquisition can be used, the sound module is an existing integrated sound card, and any integrated sound card on the market that can meet the requirements of sound acquisition and audio playback can be used, and the specific model is not limited here;

[0006] The analysis unit executes an eye tracking program to identify the eye direction of the wearer in the image information n, and the analysis unit executes a spatial positioning program to obtain an offset of the audio-visual glasses, the offset including a spatial displacement and a rotation axis angle θ;

[0007] The analysis unit executes a feature extraction program to obtain an item block, the position of the item block and an information parameter p. The analysis unit establishes a spatial coordinate system, the spatial coordinate system takes the center of the lens of the audio-visual glasses as the origin. The analysis unit transmits the position of the item block and the information parameter p to the display module. The display module aligns and unfolds the spatial coordinate system on the lens of the audio-visual glasses. The display module projects the information parameter p to a corresponding area on the lens according to the position of the item block for viewing by the wearer. When the offset is not zero, the analysis unit executes an offset correction program to offset compensate the display position of the information parameter p on the lens. By calculating the offset of the audio-visual glasses, the item selected in the frame of the lens is offset corrected. When the audio-visual glasses move, the stability of the parameter information display can be improved to prevent the screen from shaking and causing dizziness to the wearer.

[0008] The analysis unit executes the attention analysis program to adjust the display weight k of each information parameter p. By identifying the eye direction of the wearer, it can autonomously filter out the main items that the wearer is concerned about from the lens picture, obtain valuable information, and amplify the queried parameter information to facilitate the wearer to view;

[0009] The sound module obtains the voice information emitted by the wearer and transmits it to the analysis unit. The analysis unit is integrated with a register, in which the instruction audio is pre-stored. The instruction audio includes query and clear. The analysis unit compares the voice information with the instruction audio, and executes the corresponding instruction audio when the voice information matches the instruction audio. When the instruction audio is a query, the analysis unit sends a query request to the network module, and the network module obtains the information parameters p of all object blocks. When the instruction audio is clear, the analysis unit sends a clear request to the display module, and the display module clears the information parameters p of all object blocks.

[0010] Further, when the eye tracking program is executed, the analysis unit uses the first frame of the image information n as the reference frame nf and processes the reference frame nf step by step;

[0011] The eye corner is located, and the analysis unit is based on the side length d edge The reference frame nf is divided into a plurality of first blocks by a square. The analysis unit calculates the average brightness s1 of each pixel of the reference frame nf. The analysis unit calculates the average brightness l1 of each pixel of the first block. The analysis unit subtracts the average brightness l1 from the average brightness s1 of the reference frame nf to obtain a brightness difference , brightness difference When it is greater than or equal to zero, the color of the corresponding first block turns white, and the brightness difference When it is less than zero, the color of the corresponding first image block changes to black. After the colors of all first image blocks change, the analysis unit marks the leftmost and rightmost black first image blocks as the wearer's eye corners;

[0012] The side length d of the first block edge The number of pixels occupied is determined by the proportion of the pixel brightness value of the reference frame nf. The analysis unit distinguishes all pixels in the reference frame nf. The brightness value range of the pixels is 0 to 255. When the brightness value of the pixel is equal to or greater than 128, the analysis unit marks the corresponding pixel as a bright spot. When the brightness value of the pixel is less than 128, the analysis unit marks the corresponding pixel as a dark spot. The analysis unit counts the number of all bright spots and dark spots in the reference frame nf. The analysis unit calculates the average number of pixels on the long side of the rectangle and the average number of pixels on the short side of the rectangle in the reference frame nf. The analysis unit calculates the average number of pixels on the long side of the rectangle and the average number of pixels on the short side of the rectangle according to the formula The side length d of the first block is calculated edge The corresponding number of pixels;

[0013] Eyeball positioning, the analysis unit extracts the top and bottom black first blocks from the reference frame nf, and connects the leftmost, rightmost, topmost and bottommost black first blocks end to end to obtain the eye socket area, and the analysis unit divides the eye socket area into two groups according to the side length The eye socket area is divided into a plurality of second blocks by a square of a pixel of the eye socket area, and the analysis unit calculates the average brightness s2 of the pixels in the eye socket area, and the analysis unit calculates the average brightness l2 of the pixels in each second block, and the analysis unit subtracts the average brightness l2 of the pixels in each second block from the average brightness s2 of the pixels in the eye socket area to obtain a brightness difference , brightness difference When it is greater than or equal to zero, the color of the corresponding second block turns white, and the brightness difference When it is less than zero, the color of the corresponding second image block changes to black. After the colors of all the second image blocks change, the analysis unit marks the black second image block in the eye socket area as the wearer's eyeball area;

[0014] Direction recognition: the analysis unit presets two groups of rectangular rectangles, namely the first rectangular rectangle and the second rectangular rectangle. The first rectangular rectangle is tangent to the top, bottom, left and right of the eyeball area respectively, and the eye socket area is framed by the second rectangular rectangle. The analysis unit marks the diagonal corner points of the first rectangular rectangle as the eyeball center. The second rectangular rectangle is tangent to the top, bottom, left and right of the eye socket area respectively, and the eye socket area is framed by the second rectangular rectangle. The analysis unit marks the diagonal corner points of the second rectangular rectangle as the eye socket center. The analysis unit establishes an eye coordinate system with the eye socket center as the origin. The horizontal axis and the vertical axis of the eye coordinate system are in pixels, and the processing of the reference frame nf is completed;

[0015] The analysis unit calculates in real time the coordinate α of the eye center in the image information n in the eye coordinate system. The horizontal coordinate of the coordinate α is the number of horizontal pixels of the eye center minus the pixel difference of the origin of the eye coordinate system. The vertical coordinate of the coordinate α is the number of vertical pixels of the eye center minus the pixel difference of the origin of the eye coordinate system. The analysis unit marks the coordinate α as the eye direction of the wearer.

[0016] Further, when the feature extraction program is executed, the analysis unit uses the first frame of the image information w as the processing frame wf, and the analysis unit processes the processing frame wf;

[0017] Establishing a visual coordinate system, the analysis unit selects the diagonal line connecting the circumscribed rectangle of the processing frame wf as the visual midpoint, and the analysis unit establishes the visual coordinate system with the visual midpoint as the origin;

[0018] The process of selecting the circumscribed rectangle is the same as the process of selecting the first right-angle rectangle corresponding to the eyeball area;

[0019] For object recognition, the analysis unit calculates the average brightness value s3 of the processing frame wf, and the analysis unit subtracts the brightness value s3 from the brightness value of each pixel in the processing frame wf to obtain the brightness difference , the brightness value of the pixel ranges from 0 to 255, and the brightness difference When it is greater than or equal to zero, the corresponding pixel brightness value becomes 255, and the brightness difference When it is less than zero, the corresponding pixel brightness value becomes 0. After the pixel brightness value of the processed frame wf changes, the white line in the processed frame wf is the outline of the object. The analysis unit selects the outline of the object and cuts it to obtain the object block. The analysis unit records the position of each object in the processed frame wf. The block analysis unit stops processing the processed frame wf. The number of object blocks is greater than or equal to 1.

[0020] The process of selecting the object outline is the same as the process of selecting the first right-angle rectangle corresponding to the eyeball area;

[0021] The network obtains information, and the analysis unit transmits the item image block to the network module. The network module uses the image recognition engine to search for products similar to the item image block. If similar products are found, the network module transmits the information parameter p corresponding to the product to the analysis unit. If no similar products are found, the network module transmits the result of no matching products to the analysis unit. The analysis unit deletes the corresponding item image block, and the feature extraction program terminates.

[0022] The image recognition engine is a function of searching images by image installed in existing e-commerce platforms. Its technical principle is to convert the image information uploaded by the user into a feature code using a hash algorithm, compare the feature code with the feature code in the database to calculate the similarity, and feed back the products with the highest feature code similarity to the user. In this application, the network module searches for products similar to the item blocks by accessing the data interface of the image recognition engine. Any data interface on the market that can realize the function of the image recognition engine can be used, and no specific limitation is made here.

[0023] Further, when the spatial acceleration j is not zero, the analysis unit executes a spatial positioning program, and the analysis unit decomposes the acceleration j into four components, namely, jx of the x-axis, jy of the y-axis, jz of the z-axis, and jr of the rotation axis;

[0024] The analysis unit marks the acquisition time of the acceleration module as t, the initial velocity of the x-axis as v0x, the initial velocity of the y-axis as voy, the initial velocity of the z-axis as v0z, the initial position of the x-axis as x0, the initial position of the y-axis as y0, and the initial position of the z-axis as z0;

[0025] The velocity vx in the x-axis direction is , the displacement dx in the x-axis direction is , the velocity vy in the y-axis direction is , the displacement dy in the y-axis direction is , the velocity vz in the z-axis direction is , the displacement dz in the z-axis direction is , the angle θ in the direction of the rotation axis is The analysis unit adds the x-axis displacement dx, the y-axis displacement dy, and the z-axis displacement dz to obtain the spatial displacement, and the analysis unit adds the spatial displacement and the rotation axis angle θ to obtain the offset of the audio-visual glasses;

[0026] When the spatial acceleration j is zero, the spatial positioning program is terminated.

[0027] Further, when the offset correction program is executed, the analysis unit decomposes the spatial displacement into horizontal offset, vertical offset and front-back offset, and the analysis unit decomposes the rotation axis angle θ into horizontal angle and vertical angle;

[0028] The analysis unit adds the horizontal offset to the horizontal angle to obtain a horizontal compensation value, the analysis unit adds the vertical offset to the vertical angle to obtain a vertical compensation value, and the analysis unit changes the front-to-back offset to the front-to-back compensation value;

[0029] The analysis unit sequentially subtracts the horizontal compensation value, the vertical compensation value and the front-back compensation value from the position of each item tile to obtain a correction position, and the analysis unit inputs the correction position corresponding to each item tile into the display module, and the display module moves the information parameter p corresponding to each item tile to the correction position;

[0030] When the offset is zero, the offset correction process terminates.

[0031] Furthermore, when the attention analysis program is executed, the analysis unit transmits the eye direction of the wearer to the display module, the display module projects the eye direction into a spatial coordinate system, and the analysis unit marks the item tiles located at the angle of the eye direction as primary items, and marks the remaining item tiles as secondary items;

[0032] The analysis unit presets and initializes the display weight k, and the initial value of the highlight coefficient k is 0.05. The analysis unit amplifies the information parameter p corresponding to the main item, and the calculation formula of the amplification ratio μ is: , is the initial display area occupied by the main item, t stay The duration of the overlap between the wearer's eyeball direction and the area where the main item is located, the analysis unit transmits the information parameter p corresponding to the main item to the sound module, the sound module is internally integrated with a player, and the player reads the information parameter p for the wearer to listen to;

[0033] The analysis unit counts the number of times each object tile overlaps with the eye direction, and the value of the display weight k corresponding to the object tile increases by 0.01 for each increase in the number of overlaps. When the object tile disappears from the lenses of the audio-visual glasses, the display weight k is reinitialized.

[0034] The present invention has the following beneficial effects:

[0035] 1. By identifying the direction of the wearer's eyes, it can autonomously filter out the main items that the wearer is concerned about from the lens image, obtain valuable information, and amplify the queried parameter information for the wearer to view.

[0036] 2. By calculating the offset of the audio-visual glasses, the offset of the framed objects in the lens image is corrected. When the audio-visual glasses move, the stability of the parameter information display can be improved to prevent the wearer from feeling dizzy due to image shaking.

[0037] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for describing the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0039] Figure 1 The present invention is a block diagram of a data recognition system for audio-visual glasses. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] See also Figure 1 The present invention provides a technical solution: a data recognition system for audio-visual glasses, comprising an external visual module, an acceleration module, an analysis unit, a display module, an internal visual module, a network module and a sound module, wherein the external visual module obtains image information w in front of the audio-visual glasses and transmits it to the analysis unit, the internal visual module obtains image information n of a single eyeball of the wearer and transmits it to the analysis unit, and the acceleration module obtains spatial acceleration j of the audio-visual glasses and transmits it to the analysis unit;

[0042] The analysis unit executes an eye tracking program to identify the eye direction of the wearer in the image information n, and the analysis unit executes a spatial positioning program to obtain an offset of the audio-visual glasses, wherein the offset includes a spatial displacement and a rotation axis angle θ;

[0043] The analysis unit executes a feature extraction program to obtain an item image block, the position of the item image block and an information parameter p. The analysis unit establishes a spatial coordinate system, the spatial coordinate system takes the center of the lens of the audio-visual glasses as the origin. The analysis unit transmits the position of the item image block and the information parameter p to the display module. The display module aligns and unfolds the spatial coordinate system on the lens of the audio-visual glasses. The display module projects the information parameter p to a corresponding area on the lens according to the position of the item image block for viewing by the wearer. When the offset is not zero, the analysis unit executes an offset correction program to compensate for the offset of the display position of the information parameter p on the lens.

[0044] The analysis unit executes the attention analysis procedure to adjust the display weight k of each information parameter p;

[0045] The sound module obtains the voice information sent by the wearer and transmits it to the analysis unit. The analysis unit is integrated with a register, in which the command audio is pre-stored. The command audio includes query and clear. The analysis unit compares the voice information with the command audio. When the voice information matches the command audio, the corresponding command audio is executed. When the command audio is a query, the analysis unit sends a query request to the network module, and the network module obtains the information parameter p of all item blocks. When the command audio is clear, the analysis unit sends a clear request to the display module, and the display module clears the information parameter p of all item blocks.

[0046] When the eye tracking program is executed, the analysis unit uses the first frame of the image information n as the reference frame nf and processes the reference frame nf step by step;

[0047] The eye corner is located, and the analysis unit is based on the side length d edge The reference frame nf is divided into a plurality of first blocks by a square. The analysis unit calculates the average brightness s1 of each pixel of the reference frame nf. The analysis unit calculates the average brightness l1 of each pixel of the first block. The analysis unit subtracts the average brightness l1 from the average brightness s1 of the reference frame nf to obtain a brightness difference , brightness difference When it is greater than or equal to zero, the color of the corresponding first block turns white, and the brightness difference When it is less than zero, the color of the corresponding first image block changes to black. After the colors of all first image blocks change, the analysis unit marks the leftmost and rightmost black first image blocks as the wearer's eye corners;

[0048] The side length d of the first tile edge The number of pixels occupied is determined according to the proportion of the pixel brightness value of the reference frame nf. The analysis unit distinguishes all pixels in the reference frame nf. The brightness value range of the pixels is 0 to 255. When the brightness value of the pixel is equal to or greater than 128, the analysis unit marks the corresponding pixel as a bright spot. When the brightness value of the pixel is less than 128, the analysis unit marks the corresponding pixel as a dark spot. The analysis unit counts the number of all bright spots and dark spots in the reference frame nf. The analysis unit calculates the average number of pixels on the long side of the rectangle and the number of pixels on the short side of the rectangle in the reference frame nf. , the analysis unit is based on the formula The side length d of the first block is calculated edge The corresponding number of pixels;

[0049] Eyeball positioning, the analysis unit extracts the top and bottom black first blocks from the reference frame nf, and connects the leftmost, rightmost, topmost and bottommost black first blocks end to end to obtain the eye socket area, and the analysis unit divides the eye socket area into two groups according to the side length The eye socket area is divided into a plurality of second blocks by a square of a pixel of the eye socket area, and the analysis unit calculates the average brightness s2 of the pixels in the eye socket area, and the analysis unit calculates the average brightness l2 of the pixels in each second block, and the analysis unit subtracts the average brightness l2 of the pixels in each second block from the average brightness s2 of the pixels in the eye socket area to obtain a brightness difference , brightness difference When it is greater than or equal to zero, the color of the corresponding second block turns white, and the brightness difference When it is less than zero, the color of the corresponding second image block changes to black. After the colors of all the second image blocks change, the analysis unit marks the black second image block in the eye socket area as the wearer's eyeball area;

[0050] Direction recognition: the analysis unit presets two groups of rectangular rectangles, namely the first rectangular rectangle and the second rectangular rectangle. The first rectangular rectangle is tangent to the top, bottom, left and right of the eyeball area respectively, and the eye socket area is framed by the second rectangular rectangle. The analysis unit marks the diagonal corner points of the first rectangular rectangle as the eyeball center. The second rectangular rectangle is tangent to the top, bottom, left and right of the eye socket area respectively, and the eye socket area is framed by the second rectangular rectangle. The analysis unit marks the diagonal corner points of the second rectangular rectangle as the eye socket center. The analysis unit establishes an eye coordinate system with the eye socket center as the origin. The horizontal axis and the vertical axis of the eye coordinate system are in pixels, and the processing of the reference frame nf is completed;

[0051] The analysis unit calculates in real time the coordinate α of the eye center in the image information n in the eye coordinate system. The horizontal coordinate of the coordinate α is the number of horizontal pixels of the eye center minus the pixel difference of the origin of the eye coordinate system. The vertical coordinate of the coordinate α is the number of vertical pixels of the eye center minus the pixel difference of the origin of the eye coordinate system. The analysis unit marks the coordinate α as the eye direction of the wearer.

[0052] When the feature extraction program is executed, the analysis unit uses the first frame of the image information w as the processing frame wf, and the analysis unit processes the processing frame wf;

[0053] Establishing a visual coordinate system, the analysis unit selects the diagonal line connecting the circumscribed rectangle of the processing frame wf as the visual midpoint, and the analysis unit establishes the visual coordinate system with the visual midpoint as the origin;

[0054] The process of selecting the circumscribed rectangle is the same as the process of selecting the first right-angle rectangle corresponding to the eyeball area;

[0055] For object recognition, the analysis unit calculates the average brightness value s3 of the processing frame wf, and the analysis unit subtracts the brightness value s3 from the brightness value of each pixel in the processing frame wf to obtain the brightness difference , the brightness value of the pixel ranges from 0 to 255, and the brightness difference When it is greater than or equal to zero, the corresponding pixel brightness value becomes 255, and the brightness difference When it is less than zero, the corresponding pixel brightness value becomes 0. After the pixel brightness value of the processed frame wf changes, the white line in the processed frame wf is the outline of the object. The analysis unit selects the outline of the object and cuts it to obtain the object block. The analysis unit records the position of each object in the processed frame wf. The block analysis unit stops processing the processed frame wf. The number of object blocks is greater than or equal to 1.

[0056] The process of selecting the object outline is the same as the process of selecting the first right-angle rectangle corresponding to the eyeball area;

[0057] The network obtains information, and the analysis unit transmits the item image block to the network module. The network module uses the image recognition engine to search for products similar to the item image block. If similar products are found, the network module transmits the information parameter p corresponding to the product to the analysis unit. If no similar products are found, the network module transmits the result of no matching products to the analysis unit. The analysis unit deletes the corresponding item image block, and the feature extraction program terminates.

[0058] The image recognition engine is a function of searching images by image installed in existing e-commerce platforms. Its technical principle is to convert the image information uploaded by the user into a feature code using a hash algorithm, compare the feature code with the feature code in the database to calculate the similarity, and feed back the products with the highest feature code similarity to the user. In this application, the network module searches for products similar to the item blocks by accessing the data interface of the image recognition engine. Any data interface on the market that can realize the function of the image recognition engine can be used, and no specific limitation is made here.

[0059] When the spatial acceleration j is not zero, the analysis unit executes the spatial positioning procedure, and the analysis unit decomposes the acceleration j into four components, namely, jx of the x-axis, jy of the y-axis, jz of the z-axis, and jr of the rotation;

[0060] The analysis unit marks the acquisition time of the acceleration module as t, the initial velocity of the x-axis as v0x, the initial velocity of the y-axis as v0y, the initial velocity of the z-axis as v0z, the initial position of the x-axis as x0, the initial position of the y-axis as y0, and the initial position of the z-axis as z0;

[0061] The velocity vx in the x-axis direction is , the displacement dx in the x-axis direction is , the velocity vy in the y-axis direction is , the displacement dy in the y-axis direction is , the velocity vz in the z-axis direction is , the displacement dz in the z-axis direction is , the angle θ in the direction of the rotation axis is The analysis unit adds the x-axis displacement dx, the y-axis displacement dy, and the z-axis displacement dz to obtain the spatial displacement, and the analysis unit adds the spatial displacement and the rotation axis angle θ to obtain the offset of the audio-visual glasses;

[0062] When the spatial acceleration j is zero, the spatial positioning program terminates.

[0063] When the offset correction program is executed, the analysis unit decomposes the spatial displacement into horizontal offset, vertical offset and front-back offset, and the analysis unit decomposes the rotation axis angle θ into horizontal angle and vertical angle;

[0064] The analysis unit adds the horizontal offset to the horizontal angle to obtain a horizontal compensation value, the analysis unit adds the vertical offset to the vertical angle to obtain a vertical compensation value, and the analysis unit changes the front-to-back offset to the front-to-back compensation value;

[0065] The analysis unit sequentially subtracts the horizontal compensation value, the vertical compensation value and the front-back compensation value from the position of each item image block to obtain the correction position. The analysis unit inputs the correction position corresponding to each item image block into the display module, and the display module moves the information parameter p corresponding to each item image block to the correction position.

[0066] When the offset is zero, the offset correction process terminates.

[0067] When the attention analysis program is executed, the analysis unit transmits the eye direction of the wearer to the display module, the display module projects the eye direction into the spatial coordinate system, and the analysis unit marks the item tiles located at the angle of the eye direction as the main items, and marks the remaining item tiles as the secondary items;

[0068] The analysis unit presets and initializes the display weight k, and the initial value of the highlight coefficient k is 0.05. The analysis unit amplifies the information parameter p corresponding to the main item, and the calculation formula of the amplification ratio μ is: , is the initial display area occupied by the main item, t stay The duration of the overlap between the wearer's eyeball direction and the area where the main item is located, the analysis unit transmits the information parameter p corresponding to the main item to the sound module, the sound module is internally integrated with a player, and the player reads the information parameter p for the wearer to listen to;

[0069] The analysis unit counts the number of times each object tile overlaps with the eye direction. For every increase in the number of overlaps, the display weight k corresponding to the object tile increases by 0.01. When the object tile disappears from the lenses of the audio-visual glasses, the display weight k is reinitialized.

[0070] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A data recognition system for audio-visual glasses, comprising an external vision module, an acceleration module, an analysis unit, a display module, an internal vision module, a network module and a sound module, wherein the external vision module obtains image information w in front of the audio-visual glasses and transmits it to the analysis unit, the internal vision module obtains image information n of a single eyeball of a wearer and transmits it to the analysis unit, and the acceleration module obtains spatial acceleration j of the audio-visual glasses and transmits it to the analysis unit; The analysis unit executes an eye tracking program to identify the eye direction of the wearer in the image information n, and executes a spatial positioning program to obtain an offset of the audio-visual glasses, wherein the offset includes a spatial displacement and a rotation axis angle θ; Execute a feature extraction program to obtain the item image block, the position of the item image block and the information parameter p, establish a spatial coordinate system, the spatial coordinate system takes the center of the lens of the audio-visual glasses as the origin, the analysis unit transmits the position of the item image block and the information parameter p to the display module, the display module aligns and unfolds the spatial coordinate system on the lens of the audio-visual glasses, and the display module projects the information parameter p to the corresponding area on the lens according to the position of the item image block for the wearer to view. When the offset is not zero, execute an offset correction program to compensate for the offset of the display position of the information parameter p on the lens; Perform an attention analysis procedure to adjust the display weight k of each information parameter p; The sound module obtains the voice information sent by the wearer and transmits it to the analysis unit. The analysis unit is integrated with a register, and the register pre-stores the instruction audio. The instruction audio includes query and clear. The voice information is compared with the instruction audio. When the voice information matches the instruction audio, the corresponding instruction audio is executed. When the instruction audio is query, the analysis unit sends a query request to the network module, and the network module obtains the information parameters p of all item blocks. When the instruction audio is clear, the analysis unit sends a clear request to the display module, and the display module clears the information parameters p of all item blocks. Features: When the attention analysis program is executed, the analysis unit transmits the eye direction of the wearer to the display module, the display module projects the eye direction into a spatial coordinate system, and the analysis unit marks the item tiles located at the angle of the eye direction as primary items, and marks the remaining item tiles as secondary items; Preset and initialize the display weight k, and amplify the information parameter p corresponding to the main item. The calculation formula is , is the initial display area occupied by the main item, t stay The duration of the overlap between the wearer's eyeball direction and the area where the main item is located, the analysis unit transmits the information parameter p corresponding to the main item to the sound module, the sound module is internally integrated with a player, and the player reads the information parameter p for the wearer to listen to; The number of times each item tile overlaps with the eye direction is counted. For every additional overlap, the display weight k corresponding to the item tile increases by 0.

01. When the item tile disappears from the lenses of the audio-visual glasses, the display weight k is reinitialized.

2. A data recognition system for audio-visual glasses according to claim 1, It is characterized in that When the eye tracking program is executed, the first frame of the image information n is used as the reference frame nf, and the reference frame nf is processed step by step; The eye corner is located, and the analysis unit is based on the side length d edge The reference frame nf is divided into a plurality of first blocks by a square. The analysis unit calculates the average brightness s1 of each pixel of the reference frame nf. The analysis unit calculates the average brightness l1 of each pixel of the first block. The analysis unit subtracts the average brightness l1 from the average brightness s1 of the reference frame nf to obtain a brightness difference , brightness difference When it is greater than or equal to zero, the color of the corresponding first block turns white, and the brightness difference When it is less than zero, the color of the corresponding first image block changes to black. After the colors of all first image blocks change, the analysis unit marks the leftmost and rightmost black first image blocks as the wearer's eye corners; The side length d of the first block edge The number of pixels occupied is determined according to the proportion of the pixel brightness value of the reference frame nf. The analysis unit distinguishes all pixels in the reference frame nf. The brightness value range of the pixels is 0 to 255. When the brightness value of the pixel is equal to or greater than 128, the analysis unit marks the corresponding pixel as a bright spot. When the brightness value of the pixel is less than 128, the analysis unit marks the corresponding pixel as a dark spot. The analysis unit counts the number of all bright spots and dark spots in the reference frame nf. The analysis unit calculates the average number of pixels on the long side of the rectangle and the number of pixels on the short side of the rectangle in the reference frame nf. , the analysis unit is based on the formula The side length d of the first block is calculated edge The corresponding number of pixels; Eyeball positioning, the analysis unit extracts the top and bottom black first blocks from the reference frame nf, and connects the leftmost, rightmost, topmost and bottommost black first blocks end to end to obtain the eye socket area, and the analysis unit divides the eye socket area into two groups according to the side length The eye socket area is divided into a plurality of second blocks by a square of a pixel of the eye socket area, and the analysis unit calculates the average brightness s2 of the pixels in the eye socket area, and the analysis unit calculates the average brightness l2 of the pixels in each second block, and the analysis unit subtracts the average brightness l2 of the pixels in each second block from the average brightness s2 of the pixels in the eye socket area to obtain a brightness difference , brightness difference When it is greater than or equal to zero, the color of the corresponding second block turns white, and the brightness difference When it is less than zero, the color of the corresponding second image block changes to black. After the colors of all the second image blocks change, the analysis unit marks the black second image block in the eye socket area as the wearer's eyeball area; Direction recognition: the analysis unit presets two groups of rectangular rectangles, namely the first rectangular rectangle and the second rectangular rectangle. The first rectangular rectangle is tangent to the top, bottom, left and right of the eyeball area respectively, and the eye socket area is framed by the second rectangular rectangle. The analysis unit marks the diagonal corner points of the first rectangular rectangle as the eyeball center. The second rectangular rectangle is tangent to the top, bottom, left and right of the eye socket area respectively, and the eye socket area is framed by the second rectangular rectangle. The analysis unit marks the diagonal corner points of the second rectangular rectangle as the eye socket center. The analysis unit establishes an eye coordinate system with the eye socket center as the origin, and completes the processing of the reference frame nf. The analysis unit calculates the coordinates of the eye center in the eye coordinate system in real time in the image information n , the analysis unit will coordinate Marked as the wearer's eye direction.

3. A data recognition system for audio-visual glasses according to claim 1, It is characterized in that When the feature extraction program is executed, the first frame of the image information w is used as the processing frame wf, and the analysis unit processes the processing frame wf; Establishing a visual coordinate system, the analysis unit selects the diagonal line connecting the circumscribed rectangle of the processing frame wf as the visual midpoint, and the analysis unit establishes the visual coordinate system with the visual midpoint as the origin; For object recognition, the analysis unit calculates the average brightness value s3 of the processing frame wf, and the analysis unit subtracts the brightness value s3 from the brightness value of each pixel in the processing frame wf to obtain the brightness difference , the brightness value of the pixel ranges from 0 to 255, and the brightness difference When it is greater than or equal to zero, the corresponding pixel brightness value becomes 255, and the brightness difference When it is less than zero, the corresponding pixel brightness value becomes 0, and the white line in the processing frame wf is the outline of the object. The analysis unit selects the outline of the object and cuts it to obtain the object block. The analysis unit records the position of each object in the processing frame wf, and the block analysis unit stops processing the processing frame wf. The network obtains information, and the analysis unit transmits the item image block to the network module. The network module uses the image recognition engine to search for products similar to the item image block. If similar products are found, the network module transmits the information parameter p corresponding to the product to the analysis unit. If no similar products are found, the network module transmits the result of no matching products to the analysis unit. The analysis unit deletes the corresponding item image block and the feature extraction program terminates.

4. A data recognition system for audio-visual glasses according to claim 1, It is characterized in that When the spatial acceleration j is not zero, the spatial positioning program is executed, and the analysis unit decomposes the acceleration j into four components, namely jx of the x-axis, jy of the y-axis, jz of the z-axis and jr of the rotation axis; The analysis unit marks the acquisition time of the acceleration module as t, the initial velocity of the x-axis as v0x, the initial velocity of the y-axis as v0y, the initial velocity of the z-axis as v0z, the initial position of the x-axis as x0, the initial position of the y-axis as y0, and the initial position of the z-axis as z0; The velocity vx in the x-axis direction is , the displacement dx in the x-axis direction is , the velocity vy in the y-axis direction is , the displacement dy in the y-axis direction is , the velocity vz in the z-axis direction is , the displacement dz in the z-axis direction is , the angle θ in the direction of the rotation axis is The analysis unit adds the x-axis displacement dx, the y-axis displacement dy, and the z-axis displacement dz to obtain the spatial displacement, and the analysis unit adds the spatial displacement and the rotation axis angle θ to obtain the offset of the audio-visual glasses; When the spatial acceleration j is zero, the spatial positioning program is terminated.

5. The data recognition system for audio-visual glasses according to claim 1, It is characterized in that When the offset correction program is executed, the analysis unit decomposes the spatial displacement into horizontal offset, vertical offset and front-back offset, and the analysis unit decomposes the rotation axis angle θ into horizontal angle and vertical angle; The analysis unit adds the horizontal offset to the horizontal angle to obtain a horizontal compensation value, the analysis unit adds the vertical offset to the vertical angle to obtain a vertical compensation value, and the analysis unit changes the front-to-back offset to the front-to-back compensation value; Subtract the horizontal compensation value, the vertical compensation value and the front-back compensation value from the position of each item tile in turn to obtain a corrected position, input the corrected position corresponding to each item tile into a display module, and the display module moves the information parameter p corresponding to each item tile to the corrected position; When the offset is zero, the offset correction process terminates.