A wearable camera posture monitoring and recognition system and method

Abstract

A wearable camera posture monitoring and recognition system and method, the system includes: a head-mounted wear frame, a camera, a head-mounted micro camera, which is fixedly connected to the head-mounted wear frame and is used to obtain scene video information; a projection device, which projects onto the same scene shot by the micro camera to enhance information; by adjusting the optical axis direction of the projection device, the center position of the projection pattern is located on the optical axis of the head-mounted micro camera; a prompt module, which issues a prompt when an abnormality occurs in the system; an angle perception module, which is used to obtain the angle posture information of the head-mounted micro camera, and the angle posture information includes: viewing angle θ and elevation angle ω. This system solves the problems that the head-mounted camera is prone to fail to capture or incompletely capture the operating scene, and the head movement affects the experimental scoring results. This system adopts multi-projection area technology to achieve accurate guidance of user behavior and avoid identifying other irrelevant equipment.

Description

Technical Field

[0001] The present invention relates to the field of experimental monitoring, and in particular to a wearable camera posture monitoring and recognition system and method. Background Art

[0002] The name of the Chinese invention patent: An intelligent identification system and method, patent number: CN113705518A, provides an intelligent identification system, the system includes: a first-level identification device, which is set in the operation scene, used to shoot the operation process and identify the first-level detection object, and feedback the identification result; a second-level identification device, worn on the operator, used to shoot the operation process and identify the second-level detection object, and feedback the identification result; a result acquisition device, used to obtain the identification results fed back by the first-level identification device and the second-level identification device. This application adopts a multi-level identification method that cooperates with each other, and takes the position of human hands, experimental equipment, experimental results and experimental phenomena as detection objects. Under the condition of natural human operation and no interruption of the experimental process, real-time identification of the detection object in the operation scene is achieved with a high recognition rate. At the same time, this application also provides an intelligent identification method based on the above system.

[0003] During the experimental operation test, cameras are often placed in multiple positions to obtain experimental operation videos for later scoring of the test process. The multiple positions include head-mounted cameras worn on the head. Existing head-mounted cameras have the following problems: (1) The shooting range is too flexible and uncertain. For example, the applicant found in a large number of experiments and tests that if the user shakes his head slightly unconsciously, recognition errors may occur, or if the user tilts his head slightly, the detection effect of the system will decrease; (2) the operation scene is often not within the camera range, making it impossible to recognize objects outside the shooting scene; (3) the user is not sure about the camera position and whether the camera can capture the entire scene; (4) every time an experiment is conducted, it is very unnatural to deliberately adjust the position to prevent misidentification of other irrelevant equipment (for example, in circuit experiments, equipment that has not been connected before is often misidentified); (5) a large number of statistical data from the early prototype system show that about 90% of "misidentification" (i.e., identifying one object as another object) and "no recognition (i.e., there is no object that should be recognized in the scene)" are caused by the head-mounted camera not capturing the operation scene, resulting in incomplete objects or failure to detect objects. In short, "worrying that the head-mounted camera cannot capture the operation scene" is a common problem and experience anxiety for almost all testers. Especially in the examination system, customers or users have almost "zero tolerance" for the system's misperception and misidentification. Summary of the invention

[0004] In response to the problems existing in the prior art, the present invention provides a wearable camera posture monitoring and recognition system and method. The system can intuitively feel or observe the area captured by the head-mounted micro camera through a projection device, thereby avoiding misidentification due to head shaking; if the user focuses on the experimental operation process itself and does not notice the projection pattern of the projection device, the system will automatically identify and process it; during the experimental operation, if the user's head shakes or moves abnormally, the system will not only give a warning, but also automatically enter a pause state, thereby not affecting the system's evaluation of the user's operation results, and fundamentally improving the user's operation experience.

[0005] The technical solution adopted by the present invention is as follows:

[0006] A wearable camera posture monitoring and recognition system, comprising a head-mounted wearable frame, a camera,

[0007] A head-mounted miniature camera, fixedly connected to the head-mounted wearable frame, is used to obtain scene video information;

[0008] A projection device, which projects onto the same scene photographed by the micro camera to enhance the projection information; by adjusting the optical axis direction of the projection device, the center position of the projection pattern is located on the optical axis of the head-mounted micro camera;

[0009] Prompt module, which issues a prompt when an abnormality occurs in the system;

[0010] The angle sensor is used to obtain the angle posture information of the head-mounted micro camera, and the angle posture information includes: viewing angle θ and elevation angle ω.

[0011] A wearable camera posture monitoring and recognition method comprises the following steps:

[0012] S0. Establish a database for storing the size R and style S of the projection pattern;

[0013] S1. Query the size R and style S of the projection pattern corresponding to each experimental operation step in the database DB;

[0014] S2, the projection device projects onto the experimental desktop to generate N projection patterns or projection areas;

[0015] S3, judging whether the projection pattern of the head-mounted micro camera is on the experimental desktop, if yes, receiving the input video data from the head-mounted micro camera; if no, executing step S4;

[0016] S4, determining whether the projection pattern can be detected in the head-mounted micro camera, if yes, receiving input video data from the head-mounted micro camera; if no, executing step S5;

[0017] S5, judging whether the head posture of the wearer is normal, if yes, receiving the input video data from the head-mounted micro camera; if no, executing step S6;

[0018] S6. Determine whether the wearer's head is shaking normally. If yes, receive input video data from the head-mounted micro camera; if no, identify and process the objects within each projection pattern area respectively.

[0019] The beneficial effects of the present invention are:

[0020] The system is provided with a projection device, which can clearly define the shooting range, thus solving the application problems that are easy to occur when using a head-mounted camera, such as failure to capture or incomplete shooting of the operation scene, and head movement affecting the experimental scoring results. The present invention automatically generates projection patterns of different sizes and styles according to the evaluation indicators of the operation, and adopts multi-projection area technology to achieve accurate guidance of user behavior and avoid identifying other irrelevant equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A structural schematic diagram of a wearable camera posture monitoring and recognition system of the present invention;

[0022] Figure 2 A flow chart of a wearable camera posture monitoring and recognition method of the present invention;

[0023] Figure 3 A flowchart of a method according to a fifth embodiment of the present invention;

[0024] Figure 4 A schematic diagram of a neural network structure according to a fifth embodiment of the present invention;

[0025] Figure 5 A top view of the experimental desktop and its projected partitions in Example 6 of the present invention. DETAILED DESCRIPTION

[0026] The present invention is further described below in conjunction with the accompanying drawings and embodiments: In order to clearly illustrate the technical features of this solution, the present invention is described in detail below through specific implementation methods and in conjunction with its accompanying drawings. The disclosure below provides many different embodiments or examples for realizing different structures of the present invention. In order to simplify the disclosure of the present invention, the components and settings of specific examples are described below. In addition, the present invention may repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed. It should be noted that the components illustrated in the accompanying drawings are not necessarily drawn to scale. The present invention omits the description of known components and processing techniques and processes to avoid unnecessary limitations on the present invention.

[0027] like Figure 1 As shown, the present invention provides a wearable camera posture monitoring and recognition system, which is integrated with a head-mounted micro camera (referred to as micro camera), a projection device, and an angle sensing module. The angle sensing module includes an angle sensor, which is fixed on a head-mounted wear frame (referred to as wear frame) through a connector. Among them, the head-mounted wear frame is worn on the user's head and is mainly used to support other sensing devices. In the embodiment of the present invention, the wear frame adopts a headgear. The micro camera obtains scene video information, the projection device projects enhanced projection information into the same scene seen by the micro camera, and the angle sensor (such as a gyroscope) obtains the angle posture information of the micro camera. The posture information mainly includes the viewing angle (θ) and the elevation angle (ω). The viewing angle refers to the angle between the optical axis of the head-mounted micro camera and the plane α with respect to the plane (α) perpendicular to the length direction of the laboratory table; the elevation angle refers to the angle between the optical axis of the head-mounted micro camera and the plane β with respect to the plane (β) where the laboratory table is located. The system also includes a camera, and the camera includes: an upper camera and a front camera, which are used to obtain video and photo data.

[0028] Adjust the optical axis direction of the projection device so that the center position of the projection pattern (for example, a circle with a red edge, the projection pattern of the present invention represents an area or range) is approximately located on the optical axis of the micro camera, and the experimental equipment, objects or operations within the projection pattern must be in the area that can be captured by the head-mounted micro camera. In other words, the projection pattern metaphorically represents the shooting range of the head-mounted micro camera; as the user's head moves, the projection pattern changes accordingly. This provides both behavioral constraints consistent with the user experience and visual guidance for the user's head movement and experimental observation process. The projection device of the present invention can be replaced by a laser or a laser group composed of multiple laser emitters, each of which can generate projection patterns of different sizes and styles, thereby achieving accurate guidance of user behavior and avoiding the identification of other irrelevant equipment. Preferably, the system can project projection patterns of different styles at different time steps, and these projection patterns with different semantics can assist the system in guiding or evaluating the user's operating behavior.

[0029] In the first embodiment of the present invention, a wearable camera posture monitoring and recognition system is provided, the system comprising: a head-mounted wear frame, a head-mounted micro camera (referred to as micro camera), a projection device, an angle sensing module, and a camera. The angle sensing module is fixed on the head-mounted wear frame (referred to as wear frame) through a connector. The micro camera obtains scene video information, and the projection device projects enhanced projection information into the same scene seen by the micro camera. The projection device is a laser group, the angles between the projection directions of different laser projectors and the optical axis direction of the head-mounted micro camera are equal, and the projection line of any laser projector and the optical axis line of the head-mounted micro camera are located in a plane passing through the optical axis, so that the center of the scene photographed by the head-mounted micro camera is the center of the projection pattern generated by the laser group. The shooting range of the head-mounted micro camera depends on the distance between the head and the scene object, which can be controlled by the user at will.

[0030] In the second embodiment of the present invention, a wearable camera posture monitoring and recognition system is provided, the system comprising: a head-mounted wear frame, a head-mounted micro camera (referred to as micro camera), a projection device, an angle sensing module, and a camera. The angle sensing module is fixed on the head-mounted wear frame (referred to as the wear frame) through a connector. The micro camera obtains scene video information, and the projection device projects enhanced projection information into the same scene seen by the micro camera. A rectangular projection device composed of a laser group is fixed on a flexible head-mounted bracket. During the user's experimental operation, only objects or objects within the rectangular projection area can be recognized by the head-mounted micro camera, and the size range of the rectangular projection area can be changed by adjusting the projection direction of each laser. When the user focuses on an object or object on the experimental desktop in a natural way, some related objects centered on the object or object will definitely be located within the rectangular projection area, thereby ensuring that the user's current operation scene is within the shooting range of the head-mounted micro camera. Therefore, the user can safely perform the experimental operation without worrying that the head-mounted micro camera will not be able to capture or capture the current operation scene image incompletely.

[0031] In the third embodiment of the present invention, a micro motor is used to drive the laser to rotate around an axis instead of the laser group in the first embodiment, and a head-mounted micro camera is arranged on the axis (the head-mounted micro camera will not rotate with the rotation of the connecting rod). The laser projection device is fixed on the head-mounted bracket. The main advantage of this embodiment is that the number of lasers is reduced.

[0032] In the fourth embodiment of the present invention, Figure 2 As shown, the present invention also provides a wearable camera posture monitoring and recognition method, which specifically includes the following steps:

[0033] Step 1: Build a database DB, which can query the size and style of the projection pattern according to the operation steps;

[0034] Step 2: Query the size R and style S of the projection pattern corresponding to each experimental operation step in the database DB;

[0035] Step 3: The projection device generates N (N≥1) projection patterns or projection areas on the experimental desktop;

[0036] Step 4: Determine whether the projection pattern of the head-mounted micro camera is on the experimental desktop. If so, go to step 8; otherwise, go to the next step:

[0037] Step 5: Determine whether the projection pattern can be detected in the head-mounted micro camera. If yes, go to step 8; otherwise, go to the next step:

[0038] Step 6: Determine whether the head posture is normal. If so, go to step 8; otherwise, go to the next step:

[0039] Step 7: Determine whether the head is shaking normally. If so, go to step 8; otherwise, go to step 9:

[0040] Step 8: receiving input video data from the head-mounted micro camera;

[0041] Step 9: Identify and process the objects within each projection pattern area respectively, mainly scoring and evaluating the user's experimental operation based on the perception and recognition results of the operation scene and the scoring rules.

[0042] When an abnormality occurs in the system, the method further includes processing the abnormality, specifically:

[0043] (1) The system prompts the user to pay attention to the experimental desktop by voice, text or other means;

[0044] (2) Stop receiving or processing input data from the head-mounted micro camera. The system temporarily stops all camera inputs and reminds the user to pause the experiment or restart the current operation by voice or text;

[0045] (3) The projection device uses special methods to prevent the user from continuing to operate. For example, the projection device flashes quickly; projects eye-catching colors or special patterns, etc.;

[0046] (4) Execute step S4.

[0047] The “system abnormality” includes the following situations: (1) the projection pattern is not on the desktop; (2) the projection pattern cannot be detected; (3) the head posture is not standard; (4) the head is shaking.

[0048] When an abnormal situation occurs, there are two solutions. One is to restart and receive video data, such as a voice prompt: "This experiment is invalid, please restart the experiment operation." The other is to prompt the user to restart the current step, and the system starts receiving video data at the same time.

[0049] In the present invention, size R is a generalized concept. For example, if the projection pattern is a circle, R is the radius of the circle; if the projection pattern is a rectangle, R is the length and width of the rectangle. Secondly, projection patterns of different styles can be generated according to different needs. For example, if it is a warning for operation or head behavior, a chaotic interference pattern is generated; under normal circumstances, a rectangular or circular pattern is generated so as not to affect the system's normal perception and recognition of the scene.

[0050] In the present invention, judging whether the projection pattern of the head-mounted micro camera is on the experimental desktop specifically includes:

[0051] 1. Detect the projection pattern position P (Px, Py)

[0052] Collect projection pattern samples from the image frames obtained by the upper camera and annotate these samples;

[0053] The neural network model is used to train the samples to obtain the recognition model M1 of the projection pattern;

[0054] Using the recognition model M1 to recognize the projection pattern in the scene in the image frame of the upper camera;

[0055] By acquiring the detection frame position of the projected pattern, the center point position P (Px, Py) of the pattern is further projected.

[0056] 2. Detection experimental desktop position Q (Qx, Qy):

[0057] Collect experimental desktop samples from the image frames obtained by the upper camera and annotate these samples;

[0058] The neural network model is used to train the samples and obtain the recognition model M2 of the experimental desktop;

[0059] Use the desktop recognition model M2 to recognize the experimental desktop in the scene;

[0060] The center point position Q(Qx, Qy) of the experimental desktop is further calculated by obtaining the position of the detection box (also called the annotation box or bounding box) of the experimental desktop.

[0061] 3. If satisfied

[0062] ||Px-Qx||≤||ac|| / 2 and ||Py-Qy||≤||bd|| / 2

[0063] The projection of the micro camera is on the experimental desktop; otherwise, the projection of the micro camera is not on the experimental desktop. Where a and b are the length and width of the desktop detection frame image, respectively, and c and d are the length and width of the projection pattern detection frame, respectively.

[0064] In the present invention, judging whether the projection pattern can be detected in the head-mounted micro camera specifically includes:

[0065] Collecting projection pattern samples from image frames acquired from a head-mounted micro-camera and annotating the samples;

[0066] The neural network model is used to train the samples to obtain the recognition model M3 of the projection pattern;

[0067] Recognize the projection pattern in the image frame of the head-mounted micro camera using the recognition model M3;

[0068] If there is a tag that returns a projection pattern, the detection is successful; otherwise, the detection fails.

[0069] In the embodiment of the present invention, judging whether the head posture is normal specifically includes:

[0070] If θ≥θ0 or ω≥ω0 is satisfied, the head posture is not standard; otherwise, the head posture is standard. Among them, θ0 and ω0 are empirical thresholds set in advance. The empirical thresholds of θ0 and ω0 can be determined by the "false recognition rates" of a large number of different users.

[0071] In the present invention, judging whether the head shaking is normal specifically includes:

[0072] Calculating acceleration

[0073] v_θ=(θ(t+1)-θ(t))

[0074] v_ω=(ω(t+1)-ω(t))

[0075] If v_θ≥v_θ0 or v_ω≥v_ω0 is satisfied, the user's head is shaking rapidly; otherwise, the head is shaking normally. Among them, v_θ0 and v_ω0 are preset empirical thresholds. The empirical thresholds of v_θ0 and v_ω0 can be determined by the "false recognition rates" of a large number of different users.

[0076] Since the shooting direction of the head-mounted WeChat camera is consistent with the projection direction of the projection device, there will be no occlusion problem. The operation of both hands will not affect the system's acquisition of the projection pattern (recognition depends on the brightness of the projection), which will not hinder the normal and smooth progress of the experiment.

[0077] In the fifth embodiment of the present invention, the angle sensor is an acceleration gyroscope.

[0078] In the examination system, we used the following real chemical instruments for operation: iron wire, copper wire, magnesium bar, sandpaper, crucible tongs, tweezers, lighter, alcohol lamp, wide-mouth bottle filled with dilute hydrochloric acid, beaker, etc.

[0079] Use the YOLOv5s network to detect objects in the experimental scene, such as Figure 3 As shown in the figure. First, the video data of all experimental objects in the experimental scene are collected from different perspectives and distances for frame cutting, and valid pictures are selected as data sets. The collected pictures are annotated using the LabelImg annotation tool, and the annotation information is sent to the YOLOv5 network model as a training sample for training, and finally the experimental object detection model based on YOLOv5 is obtained. During the recognition process, the YOLOv5s network recognition model automatically returns the basic information of the upper left corner position, height and width of the detection box of the identified object.

[0080] like Figure 4 As shown in the figure, a color image of size 608x608 is first subjected to the convolutional neural network of Darknet53 for image feature extraction. The Focus, CBL, CSP, and SPP modules are all for more effective image feature extraction. Then, the extracted features are divided into three feature maps of different scales according to the different convolution depths. The subsequent FPN and PAN structures will perform multi-scale feature fusion on these three feature maps of different scales (the upper feature map has deeper network layers, so it contains stronger semantic information, while the lower feature has less position information loss because it has passed fewer convolutional layers) to obtain three fused feature maps of different scales. Finally, the small-size feature map is used to detect large-size objects, and the large-size feature map is used to detect small-size objects.

[0081] In the fifth embodiment of the present invention, 6 laser emitting devices are used to form a circle, thereby generating a circular projection pattern. First, YoLov5 is used to identify each laser projection position, thereby calculating the bounding box of all projection points, and multiple operation-type test experiments such as "preparing 100 grams of 5% sodium chloride solution" can be smoothly completed. In 50 repeated experiments of different operators, the recognition accuracy rate reached 100%, completely eliminating the problem of "wrong points" and "wrong points deduction" of the system for scoring points. The system detects the bounding box in real time. If the bounding box is not on the experimental desktop, the system will suspend perception and recognition of the scene and prompt the operator to pay attention to the head posture until the bounding box is detected on the experimental desktop. In this embodiment, the laser point position or its bounding box can be detected using the camera directly above or the image in the head-mounted micro camera. During the operation, if the operator's head looks left and right or turns quickly, the system will suspend receiving video data from each channel on the one hand, and prompt the user to naturally focus on the experimental process with both eyes, pay attention to standard operations, and suggest the user to re-operate the current step, so that the system enters a normal working state.

[0082] In the sixth embodiment of the present invention, a wearable camera posture monitoring and recognition system is provided, and the system includes: a head-mounted wearing frame, a head-mounted micro camera (referred to as micro camera), a projection device, an angle sensing module, and a camera. The angle sensing module is fixed on the head-mounted wearing frame (referred to as the wearing frame) through a connector. The micro camera obtains scene video information, and the projection device projects enhanced projection information into the same scene seen by the micro camera. In this embodiment, the laser projection device uses a micro projector that is arranged on the camera directly above the desktop, which can dynamically control the projection pattern. In middle school operation experiments, the placement of experimental instruments often has clear specifications. For example, users are required to put the equipment back to its original place after use, and the operation area is required to be separated from the equipment area, etc. Figure 5As shown, the laser projection pattern can be divided into an experimental instrument area, an experimental operation area, and an experimental auxiliary area. The experimental equipment and instruments needed are placed in the experimental instrument area. The experimental operation area is the core area for experimental operation and observation. The experimental auxiliary area is where auxiliary objects such as rags are placed or where experimental records are made. By checking the items and situational awareness in different areas, the user's relevant behaviors can be identified and evaluated. For example, after the experiment is completed, if the user does not clean the experimental operation area, points can be deducted according to the scoring criteria; if the used instrument is not returned to the experimental instrument area during the experiment, it is considered an irregular operation and can be warned or deducted. It should be noted that by performing situational awareness of objects on the desktop through different partitions, behavioral analysis and operation status evaluation can be performed effectively, efficiently, and accurately. In addition, this embodiment also designs different projection patterns for different operation steps to guide users to operate. For example, the projection pattern is projected onto the object that should be operated in the next step, thereby guiding the user to operate the instrument or equipment in the next step.

[0083] In the sixth embodiment of the present invention, this embodiment can control the range of perception and recognition by changing the size of the projection pattern. In fact, during the experiment, if the projection device worn on the head is too close to the object being operated, the system may not detect the bounding box of the object and cause missed detection (i.e., a "no recognition" problem). According to the method of the present invention, the system will prompt the operator (the user and the operator in the present invention are equivalent concepts) to adjust the distance between the head and the operated object. If the object still cannot be detected within the specified time, the detection fails or the user is required to repeat the operation steps until the object is detected (Note: During the experiment, the number and name of the experimental instruments and equipment required for each step of the operation are obviously known). In other words, this embodiment can control the distance between the user's head and the scene object by adjusting the size of the projection pattern, so that the user does not need to worry about the distance between the head and the scene and affect the test results, and avoid anxiety in the operation experience.

[0084] In the sixth embodiment, six projection pattern styles are used, which represent different semantics (as shown in Table 1). Table 1 shows that the system projects projection patterns of different styles at different time steps, and these projection patterns with different semantics can assist the system in guiding or evaluating the user's operation behavior. For example, when the user observes a single rectangular pattern, the user will concentrate on observing the experimental phenomenon and record the observed experimental phenomenon. For another example, when the projection pattern is projected near the balance vernier, the system will automatically read the vernier scale.

[0085] Table 1 Semantics of several projection patterns in the embodiments of the present invention

[0086]

[0087]

[0088] In the seventh embodiment of the laboratory preparation of oxygen of the present invention, the main process of the experimental operation test of the present invention is as follows:

[0089] Step 1: Check the air tightness of the oxygen generator.

[0090] A red “·”-shaped projection pattern is projected onto the experimental desktop through a head-mounted micro-projection device. When the user inserts the rubber stopper into the mouth of the test tube, the projection pattern is projected onto the rubber stopper and the vicinity of the mouth of the test tube. The micro-camera device bound to the micro-projection device captures the scene image centered on the projection pattern, and the YOLOv5 network further identifies the rubber stopper, test tube and human hand in the scene; when the user holds the test tube in one hand and presses the catheter into the water with the other hand and observes whether bubbles are generated, the “·”-shaped projection pattern falls on the human hand and the water surface successively. The micro-camera device obtains the image near the projection point and identifies the bubbles.

[0091] During the operation of this step, when the user inserts the rubber stopper into the mouth of the test tube, if the head shakes irregularly and the red "·"-shaped projection pattern is not near the mouth of the test tube, the operation image of the rubber stopper being inserted into the test tube will not be within the shooting range of the micro camera. In this case, the system will obtain the position of the "·" in the image and the position and size of the experimental desktop through the YOLOv5 network. If the YOLOv5 network cannot detect the "·" from the image, or the detected "·" is not on the experimental desktop, the system will suspend the input of all visual devices on the one hand, and prompt the user to pay attention to observe the experimental process, standardize head movement, and further prompt the user to restart the airtightness check on the other hand.

[0092] Similarly, when the user is observing the bubbles on the water surface, if the user looks around, they often cannot obtain the correct observation results. Therefore, the user is required to adjust the head posture to project the "·"-shaped projection pattern onto the water surface above the catheter buried under the water surface, so as to ensure that as long as the user follows the natural experimental process, the system will not "deduct points incorrectly" or "give points randomly".

[0093] Step 2: Weigh 3g of potassium permanganate. First, we need to evaluate the scoring point of "object on the left and weight on the right". In this step, set the micro-projection device to project a rectangular pattern. When the rectangular pattern frames the object in the balance tray, the system first identifies the position and length and width information of the rectangular pattern, balance tray, etc. through the YOLOv5 network, and then further identifies the object in the rectangular pattern frame, thereby identifying "potassium permanganate" and "weight" and their placement.

[0094] During the process of placing the weights, if the rectangular projection pattern does not frame the balance tray, the system will prompt the user to adjust the head posture; if it is still not adjusted within the set time, the micro-projection device will project a high-frequency flashing "X"-shaped pattern onto the desktop and voice prompt the user "This experiment is invalid, please start the experiment again."

[0095] Similarly, in the subsequent balance leveling process, the user is required to look at the slider to ensure that the slider is within the rectangular pattern. The YOLOv5 network recognizes the scale position of the beam within the rectangular pattern in real time. If the user's head shakes quickly and the rectangular projection pattern is not on the experimental table or cannot frame the scale position area where the slider is located, the system will prompt the user to pay attention to the leveling process; if it is still not corrected within the specified time, the system will consider the user's operation process as non-standard and lose the score for this knowledge point.

[0096] Step 3: Pour potassium permanganate into the test tube. In this step, the knowledge point to be tested is "one horizontal, two placed, three upright", that is, first place the test tube horizontally, then put potassium permanganate into the test tube, and then slowly stand the test tube upright. In this step, a circular projection pattern is used. The system will only work normally when the circular projection pattern encircles the test tube and the medicine; otherwise, the system will repeatedly remind the user to adjust the distance and direction between the head and the test tube so that the test tube and the medicine are both within the circular projection pattern.

[0097] The other steps are monitored and identified using similar methods as above.

[0098] The last step is to tidy up the test bench. In this step, multiple projection patterns are projected onto the desktop: the operation area and the instrument area. First, the YOLOv5 network is used to identify the position and size of different areas; then, the objects in each projection area are identified to determine whether they are consistent with the initial state. If they are consistent, points are added according to the scoring criteria; otherwise, points are deducted according to the scoring criteria.

[0099] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the wearable camera posture monitoring and recognition method are implemented.

[0100] A wearable camera posture monitoring and recognition device, wherein the device is applicable to the wearable camera posture monitoring and recognition system.

[0101] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A wearable camera posture monitoring and recognition method, characterized in that: The steps include: S0. Establish a database for storing the size R and style S of the projection pattern; S1. Query the size R and style S of the projection pattern corresponding to each experimental operation step in the database; S2, the projection device projects onto the experimental desktop to generate N projection patterns or projection areas; S3, judging whether the projection pattern of the head-mounted micro camera is on the experimental desktop, if yes, receiving the input video data from the head-mounted micro camera; if no, executing step S4; S4, determining whether the projection pattern can be detected in the head-mounted micro camera, if yes, receiving input video data from the head-mounted micro camera; if no, executing step S5; S5, judging whether the head posture of the wearer is normal, if yes, receiving the input video data from the head-mounted micro camera; if no, executing step S6; S6, determining whether the wearer's head is shaking normally, if yes, receiving input video data from the head-mounted micro camera, and identifying and processing the objects within each projection pattern area respectively; otherwise, performing abnormal processing; The specific method for judging whether the projection pattern of the head-mounted micro camera is on the experimental desktop is as follows: Collect projection pattern samples and experimental desktop samples in the image frame obtained by the camera, and mark the samples; The neural network model is used to train the labeled samples to obtain the recognition model M1 of the projection pattern and the recognition model M2 of the experimental desktop; The projection pattern obtained by the camera is recognized by the recognition model M1, and the experimental desktop is recognized by the recognition model M2 to obtain the monitoring frame position of the projection pattern and the detection frame position of the experimental desktop; Obtain the center point position P (Px, Py) of the projection pattern and the center point position Q (Qx, Qy) of the experimental tabletop; If satisfied ||Px-Qx||≤||ac|| / 2 and ||Py-Qy||≤||bd|| / 2 The projection of the projection device is located on the experimental desktop. Otherwise, the projection of the projection device is not on the experimental desktop and the system is abnormal. Wherein, a and b are the length and width of the desktop detection frame image, respectively, and c and d are the length and width of the projection pattern detection frame, respectively.

2. The wearable camera posture monitoring and recognition method according to claim 1, characterized in that: When an exception occurs in the system, the method further includes automatically handling the exception, specifically including: The system's prompt module prompts the user to pay attention to the experimental desktop; The system stops receiving or processing input data from the head-mounted micro camera, and the prompt module prompts the user to pause the experiment or redo the current operation.

3. The wearable camera posture monitoring and recognition method according to claim 1, characterized in that: The specific method for determining whether the projection pattern can be detected in the head-mounted micro camera is as follows: Collecting projection pattern samples from image frames acquired by a head-mounted micro camera and annotating the samples; A neural network is used to train the labeled samples to obtain a recognition model M3 of the projection pattern; Recognize the projection pattern in the image frame of the head-mounted micro camera using the recognition model M3; If the label of the projected pattern is returned, the detection is successful; Otherwise, the detection fails.

4. The wearable camera posture monitoring and recognition method according to claim 1, characterized in that: The specific method for judging whether the wearer's head posture is standard is as follows: Preset thresholds θ0 and ω0 of the viewing angle θ and the elevation angle ω; If θ≥θ0 or ω≥ω0, the wearer's head posture is not standard; otherwise, the wearer's head posture is standard.

5. The wearable camera posture monitoring and recognition method according to claim 1, characterized in that: The specific method for judging whether the wearer's head is shaking normally is as follows: Preset the wearer's head velocity thresholds v_θ0 and v_ω0; If the speeds v_θ and v_ω satisfy v_θ≥v_θ0 or v_ω≥v_ω0, the wearer's head shakes rapidly; otherwise, the wearer's head shakes normally; The calculation method of the speed v_θ, v_ω is: v_θ=(θ(t+1)-θ(t)) v_ω=(ω(t+1)-ω(t)) The angle sensing module provides data θ(t+1), θ(t), ω(t+1), ω(t).

6. The wearable camera posture monitoring and recognition method according to claim 1, characterized in that: Objects within each projection pattern area are identified and processed separately. Specifically, the user's experimental operation is scored and evaluated based on the perception of the operation scene, the recognition result, and the scoring rules.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the wearable camera posture monitoring and recognition method as described in any one of claims 1-6 are implemented.

Images