Interactive control method and system of intelligent experimental cabin based on image recognition

Abstract

The present invention relates to the technical field of experimental equipment control, and specifically discloses an intelligent experimental cabin interactive control method and system based on image recognition, comprising: controlling a 3D naked-eye holographic device to display a 3D holographic video of an experimental simulation; when a user performs an experimental operation on an interactive board, collecting an interactive image of the interactive board through a camera worn by the user; identifying a target operation trajectory of the user according to the interactive image, and updating the 3D holographic video displayed by the 3D naked-eye holographic device based on the target operation trajectory and a preset experimental logic; by setting a transparent interactive board and a camera, the user does not need to simulate the experimental operation through a handle or a touch screen, and the user's operation on an experimental object in the 3D holographic video on the interactive board through the camera is collected and recognized to update the 3D holographic video, which is closer to the direct operation of the user's hand on the experimental object in a real experimental scene, thereby improving the authenticity and user experience of the simulated experiment.

Description

Technical Field

[0001] The present invention belongs to the technical field of experimental equipment control, and in particular relates to an intelligent experimental cabin interactive control method and system based on image recognition. Background Art

[0002] At present, in order to cope with the problems of scarce experimental equipment and high experimental costs, intelligent experimental cabins have been developed, such as Figure 1 As shown, the intelligent experiment cabin includes a 3D naked-eye holographic device 1 and an operating unit 2 for simulation. The 3D naked-eye holographic device 1 is provided with a 3D imaging area 11 for presenting the experimental process. The operating unit 2 is mostly a handle, a touch screen, etc. During the experimental simulation, the user operates through the handle and the touch screen, and a 3D video of the operation process is presented in the 3D imaging area.

[0003] However, when performing experimental simulation operations through a handle or touch screen, the object operated by the user's hand is the handle or touch screen, and the object operated by the hand is not the experimental object of the experimental scene in the 3D imaging area. The user needs to observe the 3D imaging area while watching the handle or touch screen to simulate the experiment, and cannot restore the user's hand directly operating the experimental object in the real experimental scene, and the user's simulation experiment experience is poor. Summary of the invention

[0004] The purpose of the embodiments of the present invention is to provide an interactive control method and system for an intelligent experiment cabin based on image recognition, aiming to solve the problem that the current intelligent experiment cabin is operated through a handle or a touch screen and cannot simulate the user's direct hand operation of the experimental object in a real experimental scenario.

[0005] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0006] An interactive control method for an intelligent experiment cabin based on image recognition, wherein the intelligent experiment cabin comprises a 3D naked-eye holographic device, a camera and a transparent interactive board, wherein the interactive board is connected to the 3D naked-eye holographic device and is located at a viewing port of the 3D naked-eye holographic device, and a user views a 3D holographic video displayed in the 3D naked-eye holographic device through the interactive board, and the camera communicates with the 3D naked-eye holographic device, and specifically comprises the following steps:

[0007] Controlling the 3D naked-eye holographic device to display a 3D holographic video of an experimental simulation, wherein the displayed 3D holographic video displays a plurality of experimental objects;

[0008] When the user performs experimental operations on the interactive board, the interactive image of the interactive board is collected by a camera worn by the user;

[0009] Identify the target operation trajectory of the user according to the interactive image, where the target operation trajectory is the trajectory of the user's experimental operation on the interactive board mapped to the 3D holographic video;

[0010] The 3D holographic video displayed by the 3D naked-eye holographic device is updated based on the target operation trajectory and the preset experimental logic.

[0011] As a further limitation of the technical solution of the embodiment of the present invention, the interactive board is provided with a calibration mark, and before controlling the 3D naked-eye holographic device to display the 3D holographic video simulated in the experiment, the following steps are specifically included:

[0012] Controlling the 3D naked-eye holographic device to display a calibration object;

[0013] After prompting the user to align the eyes, the calibration mark and the calibration object in a straight line, controlling the camera worn by the user to capture a calibration image of the calibration object through the interactive board;

[0014] The camera is calibrated using the calibration image and the internal parameters of the camera to obtain a first coordinate transformation matrix from the camera coordinate system to the user's eye coordinate system.

[0015] As a further limitation of the technical solution of the embodiment of the present invention, when a user performs an experimental operation on the interactive board, an interactive image is collected from the interactive board through a camera worn by the user, specifically including the following steps:

[0016] When the user performs experimental operations on the interactive board, the camera worn by the user collects images of the interactive board according to a preset period to obtain an interactive image sequence.

[0017] As a further limitation of the technical solution of the embodiment of the present invention, identifying the target operation trajectory of the user according to the interactive image specifically includes the following steps:

[0018] Inputting the interactive image into a preset experimental action recognition model to identify a first operation trajectory of the user, where the first operation trajectory is a trajectory in the camera coordinate system;

[0019] Using the first coordinate conversion matrix to convert the first operation trajectory into a second operation trajectory in the eye coordinate system of the user;

[0020] The second operation trajectory is converted into an operation trajectory in the coordinate system of the 3D naked-eye holographic device by using a preset second coordinate conversion matrix of the interactive board and the 3D naked-eye holographic device, so as to obtain a target operation trajectory.

[0021] As a further limitation of the technical solution of the embodiment of the present invention, updating the 3D holographic video displayed by the 3D naked-eye holographic device based on the target operation trajectory and the preset experimental logic specifically includes the following steps:

[0022] Inputting the target operation trajectory into a preset operation category recognition model to obtain the category of the target operation trajectory;

[0023] When the category of the target operation track is a selection operation, determining an experimental object corresponding to the target operation track in the 3D holographic video as a target operation object;

[0024] Adjusting the color of the target operation object to a preset color in the 3D holographic video;

[0025] When the category of the target operation track is an experimental logic operation, determining a target operation object, and obtaining the experimental logic of the target operation object;

[0026] The 3D holographic video is updated according to the experimental logic.

[0027] As a further limitation of the technical solution of the embodiment of the present invention, when the category of the target operation trajectory is a selection operation, determining the experimental object corresponding to the target operation trajectory in the 3D holographic video as the target operation object specifically includes the following steps:

[0028] When the category of the target operation track is a selection operation, determining a target position of the target operation track in the 3D holographic video;

[0029] Determining whether there are multiple experimental objects at the target position in the 3D holographic video;

[0030] If so, in response to a selection operation for the plurality of experimental objects, determining a target operation object from the plurality of experimental objects;

[0031] If not, the experimental object corresponding to the target position is determined as the target operation object.

[0032] As a further limitation of the technical solution of the embodiment of the present invention, the interactive board is provided with a sensor, and the interactive board communicates wirelessly with the camera, and specifically further includes the following steps:

[0033] When the sensor on the interactive board detects that the user operates on the interactive board, the interactive board sends a first signal to the camera, and when the camera receives the first signal, it powers on and captures an interactive image of the interactive board;

[0034] When the sensor on the interactive board does not detect the user's operation on the interactive board within a preset time period, the interactive board sends a second signal to the camera;

[0035] The camera goes into sleep mode when receiving the second signal, so as to stop collecting interactive images of the interactive board.

[0036] As a further limitation of the technical solution of the embodiment of the present invention, controlling the 3D naked-eye holographic device to display the 3D holographic video simulated in the experiment specifically includes the following steps:

[0037] An experimental subject selected by a user is received, and an initial 3D holographic video matching the experimental subject is loaded.

[0038] An intelligent experiment cabin interactive control system based on image recognition, the intelligent experiment cabin comprises a 3D naked-eye holographic device, a camera and a transparent interactive board, the interactive board is connected to the 3D naked-eye holographic device and is located at the viewing port of the 3D naked-eye holographic device, a user views the 3D holographic video displayed in the 3D naked-eye holographic device through the interactive board, the camera communicates with the 3D naked-eye holographic device, and the intelligent experiment cabin interactive control system based on image recognition specifically comprises the following units:

[0039] a 3D holographic video display unit, used to control the 3D naked-eye holographic device to display a 3D holographic video of an experimental simulation, wherein the displayed 3D holographic video displays a plurality of experimental objects;

[0040] An interactive image acquisition unit, used for acquiring interactive images of the interactive board through a camera worn by the user when the user performs experimental operations on the interactive board;

[0041] A target operation trajectory recognition unit, configured to recognize the target operation trajectory of the user according to the interactive image, wherein the target operation trajectory is a trajectory of the user's experimental operation on the interactive board mapped to the 3D holographic video;

[0042] A 3D holographic video updating unit is used to update the 3D holographic video displayed by the 3D naked-eye holographic device based on the target operation trajectory and a preset experimental logic.

[0043] As a further limitation of the technical solution of the embodiment of the present invention, the interactive board is provided with a calibration mark, and specifically further includes the following units:

[0044] A calibration object display unit, used to control the 3D naked-eye holographic device to display a calibration object;

[0045] A calibration image acquisition unit, configured to control a camera worn by the user to acquire a calibration image of the calibration object through the interactive board after prompting the user to align the eyes, the calibration mark and the calibration object in a straight line;

[0046] A calibration unit is used to calibrate the camera using the calibration image and the internal parameters of the camera to obtain a first coordinate transformation matrix from the camera coordinate system to the user's eye coordinate system.

[0047] As a further limitation of the technical solution of the embodiment of the present invention, the interactive image acquisition unit specifically includes the following modules:

[0048] The image sequence acquisition module is used to acquire images of the interactive board according to a preset period through a camera worn by the user when the user performs experimental operations on the interactive board to obtain an interactive image sequence.

[0049] As a further limitation of the technical solution of the embodiment of the present invention, the target operation trajectory recognition unit specifically includes the following modules:

[0050] A first operation trajectory recognition module, used for inputting the interactive image into a preset experimental action recognition model to recognize a first operation trajectory of the user, where the first operation trajectory is a trajectory in the camera coordinate system;

[0051] A first operation trajectory conversion module, configured to convert the first operation trajectory into a second operation trajectory in the user's eye coordinate system by using the first coordinate conversion matrix;

[0052] The second operation trajectory conversion module is used to convert the second operation trajectory into an operation trajectory in the coordinate system of the 3D naked-eye holographic device by using the preset second coordinate conversion matrix of the interactive board and the 3D naked-eye holographic device to obtain a target operation trajectory.

[0053] As a further limitation of the technical solution of the embodiment of the present invention, the 3D holographic video update unit specifically includes the following modules:

[0054] An operation category recognition module, used for inputting the target operation trajectory into a preset operation category recognition model to obtain the category of the target operation trajectory;

[0055] An operation object determination module, configured to determine, when the category of the target operation trajectory is a selection operation, an experimental object corresponding to the target operation trajectory in the 3D holographic video as a target operation object;

[0056] A color adjustment module, used for adjusting the color of the target operation object to a preset color in the 3D holographic video;

[0057] An experimental logic determination module, used to determine a target operation object and obtain the experimental logic of the target operation object when the category of the target operation track is an experimental logic operation;

[0058] A video update module is used to update the 3D holographic video according to the experimental logic.

[0059] As a further limitation of the technical solution of the embodiment of the present invention, the operation object determination module specifically includes the following submodules:

[0060] A target position determination submodule, used to determine a target position of the target operation track in the 3D holographic video when the category of the target operation track is a selection operation;

[0061] The experimental object judgment submodule is used to judge whether there are multiple experimental objects at the target position in the 3D holographic video. If so, the target operation object selection submodule is executed; if not, the target operation object determination submodule is executed;

[0062] A target operation object selection submodule is used to respond to the selection operation on multiple experimental objects and determine the target operation object from the multiple experimental objects;

[0063] The target operation object determination submodule is used to determine the experimental object corresponding to the target position as the target operation object.

[0064] As a further limitation of the technical solution of the embodiment of the present invention, the interactive board is provided with a sensor, and the interactive board communicates wirelessly with the camera, and specifically further includes the following units:

[0065] A first signal sending unit, configured to cause the interactive board to send a first signal to the camera when a sensor on the interactive board detects that a user is operating on the interactive board, and the camera is powered on and collects an interactive image of the interactive board when receiving the first signal;

[0066] A second signal sending unit, configured to cause the interactive board to send a second signal to the camera when the sensor on the interactive board does not detect any user operation on the interactive board within a preset time period;

[0067] The image acquisition stopping unit is used to put the camera into sleep mode when receiving the second signal, so as to stop acquiring interactive images for the interactive board.

[0068] As a further limitation of the technical solution of the embodiment of the present invention, the 3D holographic video display unit specifically includes the following modules:

[0069] The initial video display module is used to receive the experimental subject selected by the user and load the initial 3D holographic video matching the experimental subject.

[0070] Compared with the prior art, the present invention has the following beneficial effects:

[0071] In the intelligent experiment cabin of the embodiment of the present invention, the interactive board is connected to the 3D naked-eye holographic device and is located at the viewing port of the 3D naked-eye holographic device. The user views the 3D holographic video displayed in the 3D naked-eye holographic device through the interactive board. After controlling the 3D naked-eye holographic device to display the 3D holographic video of the experimental simulation, when the user performs experimental operations on the interactive board, the interactive board is captured by a camera worn by the user. The user's target operation trajectory is identified according to the interactive image. The target operation trajectory is the trajectory of the user's experimental operation on the interactive board mapped to the 3D holographic video. The image displayed by the 3D naked-eye holographic device is updated based on the target operation trajectory and the preset experimental logic. 3D holographic video, since the interactive board is connected to the 3D naked-eye holographic device and is located at the viewing port of the 3D naked-eye holographic device, the user operates the experimental object in the displayed 3D holographic video on the transparent interactive board through the air, and the interactive image captured by the camera on the interactive board can identify the user's target operation trajectory, thereby updating the 3D holographic video. When doing simulation experiments, the user does not need to operate through the handle or touch screen. The camera captures the image of the interactive board and identifies the user's operation trajectory before updating the 3D holographic video. This is closer to the real experimental scene where the user directly operates the experimental object seen by hand, which improves the authenticity of the simulation experiment and improves the user's experience of the simulation experiment. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention.

[0073] Figure 1 A schematic diagram of an intelligent experimental cabin in the prior art;

[0074] Figure 2 A flow chart of an intelligent experiment cabin interactive control method based on image recognition provided by an embodiment of the present invention is shown.

[0075] Figure 3 The overall schematic diagram and the exploded structural schematic diagram of the experimental cabin of an embodiment of the present invention are shown;

[0076] Figure 4 A schematic diagram showing the user's eyes and the camera's viewing angle;

[0077] Figure 5 An application architecture diagram of an intelligent experimental cabin interactive control system based on image recognition provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0078] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0079] like Figure 1 The figure shows an experimental cabin in the prior art, which includes a 3D naked-eye holographic device 1 and an operating unit 2, wherein the 3D naked-eye holographic device includes a 3D imaging area 11, and the user performs experimental simulation operations through the operating unit 2 (handle or touch screen) to display a 3D holographic video 13 of the experimental process in the 3D imaging area 11. During the experimental simulation, the user observes the 3D holographic video 13 in the 3D imaging area 11 through his eyes. In an example, the 3D holographic video 13 displays experimental objects such as an alcohol lamp, a test tube, and a test tube clamp. At the same time, the experimental operation is performed by operating the operating unit 2 by hand, so that the 3D holographic video 13 is simulated and updated according to the user's operation. Since the user performs the experimental simulation operation through the operating unit 2 (handle or touch screen), the direct object of the user's hand operation is the handle or the touch screen, and it is impossible to restore the user's hand directly operating the experimental object in the real experimental scene, and the user's simulated experiment experience is poor.

[0080] In order to solve the above technical problems, the embodiments of the present invention provide an intelligent experiment cabin interactive control method and system based on image recognition, so that the intelligent experiment cabin simulated experiment interaction is closer to the real experiment scene where the user directly operates the experimental objects seen by hand, thereby improving the authenticity of the simulated experiment and the user's experience of the simulated experiment.

[0081] Figure 2 A flow chart of an intelligent experiment cabin interactive control method based on image recognition provided by an embodiment of the present invention is shown.

[0082] Specifically, the intelligent experiment cabin interactive control method based on image recognition according to the embodiment of the present invention comprises the following steps:

[0083] Step S101, controlling a 3D naked-eye holographic device to display a 3D holographic video of an experimental simulation, wherein the displayed 3D holographic video displays a plurality of experimental objects.

[0084] like Figure 3 As shown, the intelligent experimental cabin of the embodiment of the present invention includes a 3D naked-eye holographic device 1, a camera 4 and a transparent interactive board 3. The interactive board 3 is connected to the 3D naked-eye holographic device 1 and is located at the viewing port of the 3D naked-eye holographic device 1. The user views the 3D holographic video 13 displayed in the 3D naked-eye holographic device 1 through the interactive board 3. The camera 4 communicates with the 3D naked-eye holographic device 1.

[0085] The 3D naked-eye holographic device 1 may be a device for displaying a 3D holographic video. The technology for displaying a 3D holographic video by the 3D naked-eye holographic device 1 is prior art. The 3D naked-eye holographic device 1 is provided with a 3D imaging area 11. The 3D holographic video presented in the 3D imaging area 11 may be viewed through the viewing port of the 3D naked-eye holographic device 1. The interactive board 3 blocks the viewing port of the 3D naked-eye holographic device 1, and the interactive board 3 is transparent, such as transparent glass or a plastic acrylic board, so that the user can view the 3D holographic video presented in the 3D imaging area 11 by the 3D naked-eye holographic device 1 through the interactive board 3. The camera 4 may It is a wearable camera. The camera 4 is worn on the user during the experimental simulation. Preferably, it can be worn on the user's head so that the camera 4 is located on the user's forehead, or is hung on the user's ear, neck, etc., so that the camera 4 can collect images of the area where the interactive board 3 is located, and the camera 4 communicates with the 3D naked-eye holographic device 1, for example, through a connecting line, or through wireless communication methods such as Bluetooth and WiFi, so that the camera 4 can transmit the collected image to the 3D naked-eye holographic device 1, and update the displayed 3D holographic video after calculating the user's hand operation in the 3D naked-eye holographic device 1.

[0086] The smart experiment cabin of this embodiment can be an experiment cabin used in the fields of teaching, scientific research, industrial training, etc., and the smart experiment cabin can load applications of various experimental subjects in the corresponding technical fields. The application is provided with experimental logic, and the application can load the corresponding experimental logic in response to the user's simulation operation to update the displayed 3D holographic video according to the experimental logic, and present the process and results of the experimental simulation through the 3D holographic video.

[0087] It should be noted that, in addition to displaying the experimental objects in the simulation experiment, the 3D holographic video can also display the controls of the application required for the simulation experiment. For example, the controls may include the experimental object library, experimental actions (such as Figure 3 lighting the alcohol lamp, adding chemicals to the test tube), setting up experimental environment data (such as Figure 3 Set the ambient temperature, alcohol lamp flame temperature, chemical boiling point, etc.) and other controls.

[0088] During the simulation experiment, the user wears a camera on his head and faces the 3D naked-eye holographic device. Since the user needs to observe the 3D holographic video displayed by the 3D naked-eye holographic device, the user's eyes cannot be blocked by the camera. The camera is in an area outside the user's eyes. Taking the camera worn on the user's forehead as an example, the image captured by the camera is different from what the user sees with his eyes, so the camera needs to be calibrated first.

[0089] Specifically, a calibration mark may be provided on the interactive board, which may be a mark point etched in the center of the interactive board. After the 3D naked-eye holographic device is powered on, the 3D naked-eye holographic device is controlled to display the calibration object. After prompting the user to align the three points of the eye, the calibration mark and the calibration object to a straight line, the camera worn by the user is controlled to capture a calibration image of the calibration object through the interactive board. The calibration image includes the calibration mark on the interactive board and the displayed calibration object. The camera is calibrated using the calibration image and the camera's internal parameters to obtain a first coordinate transformation matrix from the camera coordinate system to the user's eye coordinate system. The first coordinate transformation matrix marks the position relationship of the camera relative to the user's eye.

[0090] Exemplarily, a calibration image can be first displayed on a 3D naked-eye holographic device, where the calibration image includes a calibration object, which can be one or more black and white circular dots or other marks. A prompt sound can be broadcast while displaying the calibration image to prompt the user to move his head so that the eyes are aligned with the displayed calibration object through the calibration marks on the interactive board, that is, the user's eyes, the calibration marks on the interactive board, and the displayed calibration object are in a straight line. The camera captures the calibration image, and calibrates the camera through the calibration image to obtain a first coordinate transformation matrix from the camera coordinate system to the user's eye coordinate system. For the specific calibration, reference can be made to the method of obtaining the translation and rotation matrices of two coordinate systems when calibrating the camera in the prior art, which will not be described in detail here.

[0091] In another embodiment, a calibration device may be provided to calibrate the camera by collecting the distance between the pupil and the camera, the angle of the camera, etc. when the user is looking at the calibration image, and obtain a first coordinate transformation matrix from the camera coordinate system to the user's eye coordinate system. The first coordinate transformation matrix may include the distance from the camera to the center of the user's eye, the rotation angle of the camera relative to each coordinate axis of the user's eye coordinate system, etc. The specific calibration method may refer to the camera calibration method in the prior art and will not be described in detail here.

[0092] After calibrating the camera, the user can operate on the touch screen of the 3D naked-eye holographic device to select the experimental subject, or select the experimental subject through the host computer connected to the 3D naked-eye holographic device. Taking the experimental simulation in teaching as an example, the corresponding experimental subject can be selected by selecting the chapter of the teaching subject. After receiving the experimental subject selected by the user, the 3D naked-eye holographic device loads the initial 3D holographic video matching the experimental subject. For example, the initial 3D holographic video displays the experimental objects required for the experimental subject, such as Figure 3 As shown in , the initial 3D holographic video can display experimental objects such as alcohol lamps, test tubes, test tube clamps, chemicals, etc., waiting for the user to start the experimental simulation after operating the interactive board.

[0093] Step S102: When the user performs experimental operations on the interactive board, the interactive image of the interactive board is collected through a camera worn by the user.

[0094] Since the interactive board is transparent, and the side of the interactive board facing away from the user is a 3D holographic video displayed in a 3D naked-eye holographic device, the user can watch the 3D holographic video through the interactive board. When the user needs to perform an experimental operation, the user can perform an experimental operation on the 3D holographic video on the other side of the interactive board on the side of the interactive board facing the user. The experimental operation may be operations such as selecting an experimental object, moving an experimental object, and rotating an experimental object.

[0095] like Figure 4 As shown, the interactive board 3 is at a certain distance from the displayed 3D holographic video 13, and the user's eyes can view the 3D holographic video 13 through the interactive board 3. From the perspective of the user's eyes, the user's hands can perform experimental operations such as selection, movement, and rotation on the experimental objects in the 3D holographic video 13 on the interactive board 3, that is, the user's hands directly perform experimental operations on the experimental objects in the 3D holographic video 13. Compared with the user operating the handle or watching the image on the touch screen, this embodiment uses the interactive board to make the intelligent laboratory cabin simulated experiment interaction, which is closer to the user's direct operation of the experimental objects seen by the hand in the real experimental scene, thereby improving the authenticity of the simulated experiment and the user's experience of the simulated experiment.

[0096] like Figure 4 As shown, when the user performs experimental operations on the interactive board 3 with his hands, the camera 4 worn by the user captures images of the interactive board 3 according to a preset period to obtain an interactive image sequence. The captured images include the operating points P of the user's hands on the interactive board 3. The operating trajectory of the user's hands can be generated through the operating points P on multiple images in the interactive image sequence.

[0097] In an optional embodiment, a sensor may be provided on the interactive board, which can be used to sense the user's hand. When the sensor on the interactive board detects that the user is operating on the interactive board, the interactive board sends a first signal to the camera. When the camera receives the first signal, it powers on and collects interactive images of the interactive board. When the sensor on the interactive board does not detect that the user is operating on the interactive board within a preset time, the interactive board sends a second signal to the camera. When the camera receives the second signal, it goes into sleep mode to stop collecting interactive images of the interactive board, thereby controlling the camera to collect interactive images when the user is performing experimental operations. When the user is not detected performing experimental operations for a long time, the camera is controlled to go into sleep mode to stop collecting interactive images, thereby avoiding the camera from collecting images when the user is not performing experimental operations. On the one hand, it avoids collecting images without user hand operations and reduces the amount of image data processing. On the other hand, it can reduce the power consumption of the camera and improve the battery life of the camera.

[0098] Step S103, identifying the user's target operation trajectory according to the interactive image, where the target operation trajectory is the trajectory of the user's experimental operation on the interactive board mapped to the 3D holographic video.

[0099] In one embodiment, the interactive image can be input into a preset experimental action recognition model to identify the user's first operation trajectory, which is a trajectory in the camera coordinate system. The first operation trajectory is converted into a second operation trajectory in the user's eye coordinate system using a first coordinate conversion matrix. The second operation trajectory is converted into an operation trajectory in the coordinate system of the 3D naked-eye holographic device using a preset second coordinate conversion matrix between the interactive board and the 3D naked-eye holographic device to obtain a target operation trajectory.

[0100] Among them, the experimental action recognition model can be a pre-trained model that can recognize the user's operation trajectory from the image. For example, after multiple interactive images in the interactive image sequence are input into the experimental action recognition model, the operation trajectory of the user's hand in the interactive image sequence can be identified. The experimental action recognition model can be trained by an image sequence marked with the operation trajectory. The specific training method can refer to the training methods of various neural network models in the prior art, which will not be described in detail here.

[0101] like Figure 4 As shown, since the camera is worn on the user, the camera and the user's eyes are not in the same position, the hand captured by the camera through the interactive board 3 and the hand seen by the user's eyes in the 3D holographic video correspond to different experimental objects. Figure 4 In the figure, from the perspective of the user's eyes, the user's hand selects the alcohol lamp in the 3D holographic video, while from the perspective of the camera 4, the user's hand selects the bottom of the 3D naked-eye holographic device. In order to convert the first operation trajectory P of the hand into the trajectory of the user's eye coordinate system, the first coordinate conversion matrix from the camera coordinate system to the user's eye coordinate system can be used to convert the first operation trajectory P into a second operation trajectory. The second operation trajectory is the operation trajectory of the user's hand in the user's eye coordinate system, that is, the position and angle of the user's eye relative to the second operation trajectory are determined, and then the second operation trajectory is converted into a trajectory in the coordinate system of the 3D naked-eye holographic device through the preset second coordinate conversion matrix between the interactive board and the 3D naked-eye holographic device to obtain the target operation trajectory. The target operation trajectory is a trajectory that uses the user's eyes as the projection light source and projects the real trajectory of the user's hand experimental operation on the interactive board onto the 3D holographic video.

[0102] In another embodiment, Figure 4As shown, after determining the second operation trajectory, the second operation trajectory is equivalent to: when each trajectory point of the second operation trajectory on the interactive board is taken as the coordinate origin, the position of the user's eyes relative to each trajectory point, so each point of the hand on the interactive board and the position of the user's eyes relative to the point can be determined. After determining the relative position of the interactive board and the 3D naked-eye holographic device, the distance from the interactive board to the 3D holographic video can be determined. The connection line from the user's eyes to the operation point of the hand on the interactive board can be calculated through the principle of triangular projection. The corresponding target point in the 3D holographic video can be extended, and the trajectory composed of multiple target points is the target operation trajectory.

[0103] After determining the target operation trajectory, the target operation trajectory is a trajectory in the 3D naked-eye holographic device coordinate system. The 3D holographic video is also a video composed of multiple frame images in the 3D naked-eye holographic device coordinate system. Each point on the target operation trajectory can correspond to the 3D holographic video, thereby determining that each point on the target operation trajectory corresponds to the experimental object in the 3D holographic video.

[0104] Step S104: updating the 3D holographic video displayed by the 3D naked-eye holographic device based on the target operation trajectory and the preset experimental logic.

[0105] In this embodiment, the preset experimental logic is the logic of the experimental subject knowledge. The experimental logic of different experimental subjects is different. The experimental logic can be preset in the experimental simulation application. After the target operation trajectory is determined, the experimental logic can be matched according to the target operation trajectory, and the 3D holographic video displayed by the 3D naked-eye holographic device is updated according to the experimental logic. The experimental logic of the corresponding experimental subject is configured in the experimental simulation application. The application can be an existing known experimental simulation application, which can process and update the 3D holographic video according to the detected target operation trajectory.

[0106] In one embodiment, the target operation trajectory can be input into a preset operation category recognition model to obtain the category of the target operation trajectory. When the category of the target operation trajectory is a selection operation, the experimental object corresponding to the target operation trajectory in the 3D holographic video is determined as the target operation object, and the color of the target operation object is adjusted to a preset color in the 3D holographic video. When the category of the target operation trajectory is an experimental logic operation, the target operation object is determined, and the experimental logic of the target operation object is obtained, and the 3D holographic video is updated according to the experimental logic.

[0107] Among them, the selection operation may refer to the operation of selecting an experimental object from the displayed 3D holographic video, the experimental logic operation may be the operation on the selected experimental object after the experimental object is selected, the experimental logic operation may be operations such as rotation and movement, and the operation category recognition model may be a pre-trained model. The operation category recognition model may be trained by labeling various operation trajectories with categories and using them as training data, so that the operation category recognition model can learn the ability to recognize the categories of operation trajectories.

[0108] like Figure 4 As shown, when the user's hand selects the alcohol lamp in the air on the interactive board, the alcohol lamp, as the experimental object selected by the user, can be displayed in a preset color in the 3D holographic video, such as red. When the user selects the test tube clamp in the air on the interactive board, and further moves the test tube away from the flame of the alcohol lamp in the air on the interactive board, the test tube is removed in the 3D holographic video, and the experimental logic of this operation is obtained: the test tube is converted from a heated state to an unheated state, the chemicals in the test tube stop heating, the chemicals are converted from a boiling state to a cooling state, the test tube mouth stops outputting steam, and the temperature of the test tube decreases. The displayed 3D holographic video is updated through the above experimental logic to simulate the above experimental logic, wherein the update of the 3D holographic video can refer to the technology of generating 3D holographic videos in the prior art, which will not be described in detail here.

[0109] In an optional embodiment, when the category of the target operation trajectory is a selection operation, the target position of the target operation trajectory in the 3D holographic video is determined, and it is determined whether there are multiple experimental objects at the target position in the 3D holographic video. If so, in response to the selection operation for multiple experimental objects, the target operation object is determined from the multiple experimental objects. If not, the experimental object corresponding to the target position is determined as the target operation object.

[0110] Specifically, the user's hand operation on the interactive board is a two-dimensional operation, that is, the user's hand operation trajectory on the interactive board is a two-dimensional trajectory, and the 3D holographic video is a three-dimensional video, triggered from the user's eye perspective, the user's hand operation point on the interactive board is mapped to the target position in the 3D holographic video. There may be multiple overlapping experimental objects in the depth direction. When there are multiple experimental objects at different depths of a target position, the multiple experimental objects can be displayed separately, so that the user can select an experimental object from multiple experimental objects as the target operation object, so that the user can select experimental objects at different depths in the 3D holographic video through the two-dimensional operation on the interactive board, thereby realizing the operation on depth of field.

[0111] In the intelligent experiment cabin of the embodiment of the present invention, the interactive board is connected to the 3D naked-eye holographic device and is located at the viewing port of the 3D naked-eye holographic device. The user views the 3D holographic video displayed in the 3D naked-eye holographic device through the interactive board. After controlling the 3D naked-eye holographic device to display the 3D holographic video of the experimental simulation, when the user performs the experimental operation on the interactive board, the interactive image is collected from the interactive board through the camera worn by the user, and the user's target operation trajectory is determined according to the interactive image. The target operation trajectory is the trajectory of the user's experimental operation on the interactive board mapped to the 3D holographic video. The 3D holographic video displayed by the 3D naked-eye holographic device is updated based on the target operation trajectory and the preset experimental logic. The naked-eye holographic device is connected to and located at the viewing port of the 3D naked-eye holographic device. The user operates the experimental object in the displayed 3D holographic video on the transparent interactive board through the air. The interactive image captured by the camera on the interactive board can identify the user's target operation trajectory, thereby updating the 3D holographic video. The user does not need to operate through the handle or touch screen when doing a simulation experiment. The user operates the experimental object in the displayed 3D holographic video on the interactive board through the air. The camera captures the image of the interactive board and then identifies the user's operation trajectory and updates the 3D holographic video. This is closer to the user's direct operation of the experimental object seen by the hand in the real experimental scene, which improves the authenticity of the simulation experiment and improves the user's experience of the simulation experiment.

[0112] Figure 5 The application architecture diagram of the intelligent experiment cabin interactive control system based on image recognition provided by an embodiment of the present invention is shown. The intelligent experiment cabin interactive control system based on image recognition in this embodiment should specifically include the following units:

[0113] The intelligent experiment cabin includes a 3D naked-eye holographic device, a camera and a transparent interactive board. The interactive board is connected to the 3D naked-eye holographic device and is located at the viewing port of the 3D naked-eye holographic device. The user views the 3D holographic video displayed in the 3D naked-eye holographic device through the interactive board. The camera communicates with the 3D naked-eye holographic device. The intelligent experiment cabin interactive control system based on image recognition specifically includes the following units:

[0114] A 3D holographic video display unit 501 is used to control the 3D naked-eye holographic device to display a 3D holographic video of an experimental simulation, wherein the displayed 3D holographic video displays a plurality of experimental objects;

[0115] The interactive image acquisition unit 502 is used to acquire interactive images of the interactive board through a camera worn by the user when the user performs experimental operations on the interactive board;

[0116] A target operation trajectory recognition unit 503 is used to recognize the target operation trajectory of the user according to the interactive image, where the target operation trajectory is a trajectory of the user's experimental operation on the interactive board mapped to the 3D holographic video;

[0117] The 3D holographic video updating unit 504 is used to update the 3D holographic video displayed by the 3D naked-eye holographic device based on the target operation trajectory and the preset experimental logic.

[0118] As a further limitation of the technical solution of the embodiment of the present invention, the interactive board is provided with a calibration mark, and specifically further includes the following units:

[0119] A calibration object display unit, used to control the 3D naked-eye holographic device to display a calibration object;

[0120] A calibration image acquisition unit, configured to control a camera worn by the user to acquire a calibration image of the calibration object through the interactive board after prompting the user to align the eyes, the calibration mark and the calibration object in a straight line;

[0121] A calibration unit is used to calibrate the camera using the calibration image and the internal parameters of the camera to obtain a first coordinate transformation matrix from the camera coordinate system to the user's eye coordinate system.

[0122] As a further limitation of the technical solution of the embodiment of the present invention, the interactive image acquisition unit 502 specifically includes the following modules:

[0123] The image sequence acquisition module is used to acquire images of the interactive board according to a preset period through a camera worn by the user when the user performs experimental operations on the interactive board to obtain an interactive image sequence.

[0124] As a further limitation of the technical solution of the embodiment of the present invention, the target operation trajectory identification unit 503 specifically includes the following modules:

[0125] A first operation trajectory recognition module, used for inputting the interactive image into a preset experimental action recognition model to recognize a first operation trajectory of the user, where the first operation trajectory is a trajectory in the camera coordinate system;

[0126] A first operation trajectory conversion module, configured to convert the first operation trajectory into a second operation trajectory in the user's eye coordinate system by using the first coordinate conversion matrix;

[0127] The second operation trajectory conversion module is used to convert the second operation trajectory into an operation trajectory in the coordinate system of the 3D naked-eye holographic device by using the preset second coordinate conversion matrix of the interactive board and the 3D naked-eye holographic device to obtain a target operation trajectory.

[0128] As a further limitation of the technical solution of the embodiment of the present invention, the 3D holographic video updating unit 504 specifically includes the following modules:

[0129] An operation category recognition module, used for inputting the target operation trajectory into a preset operation category recognition model to obtain the category of the target operation trajectory;

[0130] An operation object determination module, configured to determine, when the category of the target operation trajectory is a selection operation, an experimental object corresponding to the target operation trajectory in the 3D holographic video as a target operation object;

[0131] A color adjustment module, used for adjusting the color of the target operation object to a preset color in the 3D holographic video;

[0132] An experimental logic determination module, used to determine a target operation object and obtain the experimental logic of the target operation object when the category of the target operation track is an experimental logic operation;

[0133] A video update module is used to update the 3D holographic video according to the experimental logic.

[0134] As a further limitation of the technical solution of the embodiment of the present invention, the operation object determination module specifically includes the following submodules:

[0135] A target position determination submodule, used to determine a target position of the target operation track in the 3D holographic video when the category of the target operation track is a selection operation;

[0136] The experimental object judgment submodule is used to judge whether there are multiple experimental objects at the target position in the 3D holographic video. If so, the target operation object selection submodule is executed; if not, the target operation object determination submodule is executed;

[0137] A target operation object selection submodule is used to respond to the selection operation on multiple experimental objects and determine the target operation object from the multiple experimental objects;

[0138] The target operation object determination submodule is used to determine the experimental object corresponding to the target position as the target operation object.

[0139] As a further limitation of the technical solution of the embodiment of the present invention, the interactive board is provided with a sensor, and the interactive board communicates wirelessly with the camera, and specifically further includes the following units:

[0140] A first signal sending unit, configured to cause the interactive board to send a first signal to the camera when a sensor on the interactive board detects that a user is operating on the interactive board, and the camera is powered on and collects an interactive image of the interactive board when receiving the first signal;

[0141] A second signal sending unit, configured to cause the interactive board to send a second signal to the camera when the sensor on the interactive board does not detect any user operation on the interactive board within a preset time period;

[0142] The image acquisition stopping unit is used to put the camera into sleep mode when receiving the second signal, so as to stop acquiring interactive images for the interactive board.

[0143] As a further limitation of the technical solution of the embodiment of the present invention, the 3D holographic video display unit specifically includes the following modules:

[0144] The initial video display module is used to receive the experimental subject selected by the user and load the initial 3D holographic video matching the experimental subject.

[0145] It should be understood that, although each step in the flow chart of each embodiment of the present invention is shown in sequence according to the indication of the arrow, these steps are not necessarily performed in sequence according to the order indicated by the arrow. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be performed in other orders. Moreover, at least a portion of the steps in each embodiment may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these sub-steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of other steps or sub-steps or stages of other steps.

[0146] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0147] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0148] The above embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention patent shall be subject to the attached claims.

[0149] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An interactive control method for an intelligent experimental cabin based on image recognition, characterized in that: The intelligent experiment cabin includes a 3D naked-eye holographic device, a camera and a transparent interactive board. The interactive board is connected to the 3D naked-eye holographic device and is located at the viewing port of the 3D naked-eye holographic device. The user views the 3D holographic video displayed in the 3D naked-eye holographic device through the interactive board. The camera communicates with the 3D naked-eye holographic device, and specifically includes the following steps: Controlling the 3D naked-eye holographic device to display a 3D holographic video of an experimental simulation, wherein the displayed 3D holographic video displays a plurality of experimental objects; When the user performs experimental operations on the interactive board, the interactive image of the interactive board is collected by a camera worn by the user; Identify the target operation trajectory of the user according to the interactive image, where the target operation trajectory is the trajectory of the user's experimental operation on the interactive board mapped to the 3D holographic video; updating the 3D holographic video displayed by the 3D naked-eye holographic device based on the target operation trajectory and the preset experimental logic; Identifying the target operation trajectory of the user according to the interactive image specifically includes the following steps: Inputting the interactive image into a preset experimental action recognition model to identify a first operation trajectory of the user, where the first operation trajectory is a trajectory in a camera coordinate system; The first operation track is converted into a second operation track in the eye coordinate system of the user by using a first coordinate conversion matrix, wherein the first coordinate conversion matrix is ​​a conversion matrix from a camera coordinate system to an eye coordinate system of the user; The second operation trajectory is converted into an operation trajectory in the coordinate system of the 3D naked-eye holographic device by using a preset second coordinate conversion matrix of the interactive board and the 3D naked-eye holographic device, so as to obtain a target operation trajectory.

2. The method according to claim 1, characterized in that The interactive board is provided with calibration marks. Before controlling the 3D naked-eye holographic device to display the 3D holographic video simulated in the experiment, the following steps are specifically included: Controlling the 3D naked-eye holographic device to display a calibration object; After prompting the user to align the eyes, the calibration mark and the calibration object in a straight line, controlling the camera worn by the user to capture a calibration image of the calibration object through the interactive board; The camera is calibrated using the calibration image and the internal parameters of the camera to obtain a first coordinate transformation matrix from the camera coordinate system to the user's eye coordinate system.

3. The method according to claim 1, characterized in that When the user performs experimental operations on the interactive board, the interactive image is collected from the interactive board through a camera worn by the user, which specifically includes the following steps: When the user performs experimental operations on the interactive board, the camera worn by the user collects images of the interactive board according to a preset period to obtain an interactive image sequence.

4. The method according to any one of claims 1 to 3, characterized in that: Updating the 3D holographic video displayed by the 3D naked-eye holographic device based on the target operation trajectory and the preset experimental logic specifically includes the following steps: Inputting the target operation trajectory into a preset operation category recognition model to obtain the category of the target operation trajectory; When the category of the target operation track is a selection operation, determining an experimental object corresponding to the target operation track in the 3D holographic video as a target operation object; Adjusting the color of the target operation object to a preset color in the 3D holographic video; When the category of the target operation track is an experimental logic operation, determining a target operation object, and obtaining the experimental logic of the target operation object; The 3D holographic video is updated according to the experimental logic.

5. The method according to claim 4, characterized in that When the category of the target operation track is a selection operation, determining the experimental object corresponding to the target operation track in the 3D holographic video as the target operation object specifically includes the following steps: When the category of the target operation track is a selection operation, determining a target position of the target operation track in the 3D holographic video; Determining whether there are multiple experimental objects at the target position in the 3D holographic video; If so, in response to a selection operation for the plurality of experimental objects, determining a target operation object from the plurality of experimental objects; If not, the experimental object corresponding to the target position is determined as the target operation object.

6. The method according to any one of claims 1 to 3, characterized in that: The interactive board is provided with a sensor, and the interactive board communicates wirelessly with the camera, and specifically includes the following steps: When the sensor on the interactive board detects that the user operates on the interactive board, the interactive board sends a first signal to the camera, and when the camera receives the first signal, it powers on and captures an interactive image of the interactive board; When the sensor on the interactive board does not detect the user's operation on the interactive board within a preset time period, the interactive board sends a second signal to the camera; The camera goes into sleep mode when receiving the second signal, so as to stop collecting interactive images of the interactive board.

7. The method according to any one of claims 1 to 3, characterized in that: Controlling the 3D naked-eye holographic device to display the 3D holographic video simulated by the experiment specifically comprises the following steps: An experimental subject selected by a user is received, and an initial 3D holographic video matching the experimental subject is loaded.

8. An intelligent experimental cabin interactive control system based on image recognition, characterized in that: The intelligent experiment cabin includes a 3D naked-eye holographic device, a camera and a transparent interactive board. The interactive board is connected to the 3D naked-eye holographic device and is located at the viewing port of the 3D naked-eye holographic device. The user views the 3D holographic video displayed in the 3D naked-eye holographic device through the interactive board. The camera communicates with the 3D naked-eye holographic device. The intelligent experiment cabin interactive control system based on image recognition specifically includes the following units: a 3D holographic video display unit, used to control the 3D naked-eye holographic device to display a 3D holographic video of an experimental simulation, wherein the displayed 3D holographic video displays a plurality of experimental objects; An interactive image acquisition unit, used for acquiring interactive images of the interactive board through a camera worn by the user when the user performs experimental operations on the interactive board; A target operation trajectory recognition unit, configured to recognize the target operation trajectory of the user according to the interactive image, wherein the target operation trajectory is a trajectory of the user's experimental operation on the interactive board mapped to the 3D holographic video; A 3D holographic video updating unit, configured to update the 3D holographic video displayed by the 3D naked-eye holographic device based on the target operation trajectory and a preset experimental logic; The target operation trajectory recognition unit includes the following modules: A first operation trajectory recognition module, used for inputting the interactive image into a preset experimental action recognition model to recognize a first operation trajectory of the user, where the first operation trajectory is a trajectory in a camera coordinate system; A first operation trajectory conversion module, configured to convert the first operation trajectory into a second operation trajectory in the user's eye coordinate system by using a first coordinate conversion matrix, wherein the first coordinate conversion matrix is ​​a conversion matrix from a camera coordinate system to the user's eye coordinate system; The second operation trajectory conversion module is used to convert the second operation trajectory into an operation trajectory in the coordinate system of the 3D naked-eye holographic device by using the preset second coordinate conversion matrix of the interactive board and the 3D naked-eye holographic device to obtain a target operation trajectory.

9. The system according to claim 8, characterized in that It also includes the following units: A calibration object display unit, used to control the 3D naked-eye holographic device to display a calibration object; A calibration image acquisition unit, configured to control a camera worn by the user to acquire a calibration image of the calibration object through the interactive board after prompting the user to look at the calibration object; A calibration unit is used to calibrate the camera using the calibration image and the internal parameters of the camera to obtain a first coordinate transformation matrix from the camera coordinate system to the user's eye coordinate system.

Images