Remote synchronous control system based on ball head camera
By using an inertial measurement unit and a line of sight determination unit in a head-mounted display, the dome camera is driven to collect and synchronize the display of surrounding environment images, solving the problem that the head-mounted display cannot be remotely linked with the camera in the prior art, real-time image display and Controllability improves user experience.
Patent Information
- Application Number
- CN202211408920.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-11-11
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2042-11-11
AI Technical Summary
The existing head-mounted monitors cannot connect remotely with cameras that are not in the same situation, and cannot present real-time images captured by the camera to the user in real time, and cannot directly control the shooting status of the camera, resulting in the user being unable to synchronously change the camera's field of view range.
Using a remote synchronous control system based on a ball camera, the inertia measurement unit collects the action status information of the head-mounted display, generates action control commands to drive the ball camera to perform synchronous actions, collects the surrounding environment images, and sends the image data stream back to the head Wear a display, adjust the display status of the virtual image through the line of sight determination unit and the display adjustment unit, and realize the linkage control between the dome camera and the head-mounted display.
Improve the real-time and controllability of the image display of the head-mounted monitor, and users can control the dome camera in real time, improving the user experience.
Smart Images

Figure CN116347058B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of head mounted display control, and in particular to a remote synchronous control system based on a ball head camera. Background Art
[0002] Helmet-type or glasses-type head-mounted displays have become the main development form of virtual display technology. Head-mounted displays play pre-formed images to users, or superimpose pre-formed images on real environments to form VR display modes or AR display modes. Existing head-mounted displays are not able to remotely connect to cameras that are not in the same place, and present real-time images captured by the camera to users in real time. It is even more impossible to directly control the shooting status of the camera through the head-mounted display, so that users can watch real-time images in the corresponding field of view according to their actual needs. It can be seen that the existing head-mounted display cannot be remotely linked with the camera, so it is impossible to synchronously control the camera to change its own shooting status and synchronously present the real-time image captured by the camera to the user wearing the head-mounted display. Summary of the invention
[0003] In view of the defects in the prior art, the present invention provides a remote synchronous control system based on a ball head camera, which uses inertial measurement to collect action state information of a head mounted display, and uses this as an action control instruction; and uses the action control instruction to drive the ball head camera to perform corresponding synchronous actions, so that the ball head camera collects surrounding environment images during the synchronous action; then the surrounding environment images are converted into image data streams and sent back to the head mounted display, so that the head mounted display displays a virtual image consistent with the surrounding environment images to the user; the user's viewing line of sight information is collected through a line of sight determination unit, and the display state of the virtual image on the head mounted display is adjusted through a display adjustment unit according to the viewing line of sight information. The above system controls the ball head camera and the head mounted display in a linkage manner, and takes the head movement of the user when wearing the head mounted display as a reference to drive the head mounted display to move, and linkage controls the ball head camera to synchronously change its own camera state, so that the user can realize remote control of the ball head camera, and at the same time, the image captured by the ball head camera is presented to the user in real time, thereby improving the real-time and controllability of the image display of the head mounted display, and improving the user experience of the head mounted display.
[0004] The present invention provides a remote synchronous control system based on a ball head camera, which comprises:
[0005] A head mounted display for presenting virtual images to a user wearing the device;
[0006] an inertial measurement unit, which is disposed in the head-mounted display and is used to collect motion state information of the head-mounted display;
[0007] A control unit connected to the inertial measurement unit, and configured to generate an action control instruction according to the action state information;
[0008] A ball head camera, which is used to collect images of the surrounding environment;
[0009] A driving unit connected to the control unit, and configured to drive the ball head camera to perform corresponding synchronous actions according to the action control instruction, so that the ball head camera can collect images of the surrounding environment during the synchronous actions;
[0010] The control unit is also used to convert the surrounding environment image into an image data stream, and send the image data stream back to the head mounted display;
[0011] The head mounted display is further used to convert the image data stream into an image display signal, so as to display the virtual image consistent with the surrounding environment image;
[0012] A sight line determination unit, which is disposed in the head mounted display and is used to collect the viewing sight line information of the user;
[0013] A display adjustment unit is disposed in the head mounted display and is used to adjust a display state of the virtual image on the head mounted display according to the viewing line of sight information.
[0014] Further, the inertial measurement unit collects the action state information of the head mounted display specifically including:
[0015] The inertial measurement unit includes an attitude angle measurement subunit and an accelerometer unit;
[0016] The attitude angle measurement subunit is used to detect the action attitude angle information of the head mounted display when the user wears the head mounted display to perform an action; wherein the action attitude angle information includes the pitch angle and the heading angle of the head mounted display during the action;
[0017] The accelerometer unit is used to detect the motion acceleration information of the head mounted display when the user wears the head mounted display and performs an action.
[0018] Further, the control unit generates an action control instruction according to the action state information, specifically comprising:
[0019] The control unit determines the angle change range and angle change speed of the pitch angle and the heading angle of the head mounted display during the action according to the action posture angle information;
[0020] Then, according to the angle change range and the angle change speed, a first action control instruction is generated for controlling the ball head camera to adjust the angles in the pitch direction and the heading direction during the synchronous action;
[0021] The control unit generates a second motion control instruction for controlling the ball head camera to adjust the shooting focal length during the synchronous motion according to the motion acceleration information.
[0022] Furthermore, before generating the first action control instruction and the second action control instruction, the control unit further includes:
[0023] The control unit performs Kalman filtering on the motion posture angle information and the motion acceleration information, thereby achieving data smoothing processing on the motion posture angle information and the motion acceleration information.
[0024] Further, the control unit generates, according to the motion acceleration information, a second motion control instruction for controlling the ball head camera to adjust the shooting focal length during the synchronous motion, specifically including:
[0025] The control unit performs secondary integration processing on the motion acceleration information to obtain corresponding position information; and then generates the second motion control instruction based on the position information; wherein the second motion control instruction is used to adjust the shooting focal length of the ball head camera, thereby realizing the zooming in or out of the shooting picture of the ball head camera.
[0026] Furthermore, after the control unit performs secondary integration processing on the motion acceleration information, the corresponding position information obtained specifically includes:
[0027] When obtaining the position information, the AHRS algorithm is used to obtain the attitude information. In the process of using the AHRS algorithm, a gyroscope is used as the only measuring device for the current direction of motion. When the gyroscope is used to measure the direction, the gyroscope is controlled to rotate in a plane, and stops at a preset fixed angle and measures the static measurement value of the gyroscope at different fixed angles. The rotation of the gyroscope is controlled by a pulse motor, and a counter is added to the pulse motor. The counter outputs a high pulse signal after the motor rotates a full circle, and communicates with the gyroscope through a serial port to obtain the measurement value of the gyroscope and the working voltage value of the gyroscope. Before using the gyroscope to measure the direction, it is determined whether the gyroscope is working normally and whether the working voltage is normal. At the same time, the number of pulses input to the pulse motor between the rising edges of the two high pulse signals output by the counter is collected to determine whether the pulse motor is working normally. Self-test information is generated according to the working state of the gyroscope and the normal working state of the pulse motor, and the measurement start-up of the gyroscope is controlled. The process is as follows:
[0028] Step S1, using the following formula (1), according to the communication status with the gyroscope, determine whether the gyroscope is working normally and whether the working voltage is normal,
[0029]
[0030] In the above formula (1), E g represents the working state judgment value of the gyroscope; F represents the output value of the return information received from the gyroscope. If the return information of the gyroscope is received, F=1, otherwise F=0; {F=1|} represents that under the condition of F=1, the equation behind | is judged whether it is established. If it is established, the function value is 1, if not, the function value is 0, and if the condition of F=1 is not satisfied, the function value is directly set to 0; && represents the logical relationship and; A(d) represents the frame header data in the dth return information received from the gyroscope; A 0 Represents the general data frame header in the valid return data of the gyroscope; R(d) represents the frame tail data in the dth return information received from the gyroscope; R 0 Indicates the general data frame tail in the valid return data of the gyroscope; U(d) indicates the extraction of data on a specific bit in the dth return information received from the gyroscope, and the data on the specific bit is the current operating voltage of the gyroscope extracted; U 0 represents the external power supply voltage provided to the gyroscope; D represents the total number of return information received from the gyroscope; It means that if the value of d is substituted from 1 to D into the brackets, if there is one or more d values that make the equation in the brackets valid, then the overall equation is valid, otherwise, the overall equation is not valid;
[0031] If E g =1, indicating that the gyroscope's working state self-test is normal;
[0032] If E g =0, indicating that the self-check of the working state of the gyroscope is abnormal;
[0033] Step S2, using the following formula (2), according to the number of pulses input to the pulse motor between the rising edges of two high pulse signals output by the counter, to determine whether the pulse motor is working normally,
[0034]
[0035] In the above formula (2), E k represents the working state judgment value of the pulse motor; N represents the number of pulses input to the pulse motor between the rising edges of the two high pulse signals output by the counter; θ 0 Indicates the angle value of the pulse motor when a pulse signal is input; || indicates obtaining the absolute value; Indicates rounding up; Z{} indicates a judgment function. If the formula in the brackets is true, the function value is 1; if the formula in the brackets is not true, the function value is 0;
[0036] If E k =1, indicating that the pulse motor's working state self-test is normal;
[0037] If E k =0, indicating that the pulse motor's working state self-check is abnormal;
[0038] Step S3, using the following formula (3), according to the working state of the gyroscope and the normal working state of the pulse motor, generating self-test information and controlling the measurement start-up enable of the gyroscope,
[0039]
[0040] In the above formula (3), B represents the self-test information, where the information frame header is A 0 , the frame end is R 0 , self-test information data is E g and E k ; E represents the measurement start enable control value of the gyroscope;
[0041] If E=0, it means that the gyroscope self-check is normal, and the direction measurement of the gyroscope is started;
[0042] If E=1, it means that the gyroscope self-check is abnormal, then after troubleshooting and repairing according to the self-check information, the above steps S1 to S3 are performed again.
[0043] Furthermore, the driving unit drives the ball head camera to perform corresponding synchronous actions according to the action control instruction, so that the ball head camera collects the surrounding environment image during the synchronous action.
[0044] The driving unit comprises a first driving sub-unit and a second driving sub-unit;
[0045] The first driving subunit is used to drive the ball head camera to perform a rotation action in a pitch direction and a heading direction synchronized with the head mounted display according to the first action control instruction;
[0046] During the rotation of the ball head camera, the second driving subunit drives the ball head camera to change the shooting focal length according to the second action control instruction.
[0047] Further, the control unit converts the surrounding environment image into an image data stream, and sends the image data stream back to the head mounted display specifically including:
[0048] After performing image background noise reduction processing on the surrounding environment image, the control unit converts the surrounding environment image into an image data stream that matches the image playback format of the head-mounted display, and synchronously sends the image data stream back to the head-mounted display.
[0049] The head mounted display converts the image data stream into an image display signal, thereby displaying the virtual image consistent with the surrounding environment image, specifically including:
[0050] The head-mounted display generates a left-eye image display signal and a right-eye image display signal according to the image data stream, and inputs the signals to the left-eye display screen and the right-eye display screen of the head-mounted display, respectively, so as to display the left-eye image and the right-eye image, respectively, so that the head-mounted display displays a virtual three-dimensional image consistent with the surrounding environment image.
[0051] Further, the sight line determination unit collects the viewing sight line information of the user specifically including:
[0052] The sight line determination unit includes an infrared ray transmitting subunit, an infrared ray receiving subunit and a sight line calculation subunit;
[0053] The infrared emitting subunit is used to emit infrared rays to the left eye or the right eye of the user;
[0054] The infrared receiving subunit is used to receive infrared rays emitted by the infrared emitting subunit and reflected by the left eye or the right eye of the user;
[0055] The sight line calculation subunit is used to determine the real-time sight line direction of the left eye or the right eye of the user according to the infrared rays received by the infrared receiving subunit, and use this as the viewing sight line information;
[0056] The display adjustment unit adjusts the display state of the virtual image on the head mounted display according to the viewing line of sight information, specifically comprising:
[0057] The display adjustment unit is used to adjust the display position of the left-eye image on the left-eye display screen or adjust the display position of the right-eye image on the right-eye display screen according to the viewing line of sight information.
[0058] Compared with the prior art, the remote synchronous control system based on the ball head camera uses inertial measurement to collect the action state information of the head mounted display, and uses this action control instruction; and uses the action control instruction to drive the ball head camera to perform corresponding synchronous actions, so that the ball head camera collects the surrounding environment image during the synchronous action; then the surrounding environment image is converted into an image data stream and sent back to the head mounted display, so that the head mounted display displays a virtual image consistent with the surrounding environment image to the user; the user's viewing line of sight information is also collected through the line of sight determination unit, and the display state of the virtual image on the head mounted display is adjusted through the display adjustment unit according to the viewing line of sight information. The above system controls the ball head camera and the head mounted display in linkage, and takes the head movement of the user when wearing the head mounted display as a reference to drive the head mounted display to move, and linkage controls the ball head camera to synchronously change its own camera state, so that the user can realize remote control of the ball head camera, and at the same time, the image captured by the ball head camera is presented to the user in real time, thereby improving the real-time and controllability of the image display of the head mounted display, and improving the user experience of the head mounted display.
[0059] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings.
[0060] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0061] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0062] Figure 1 This is a schematic structural diagram of a remote synchronous control system based on a ball head camera provided by the present invention. DETAILED DESCRIPTION
[0063] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0064] See also Figure 1 , is a schematic diagram of the structure of a remote synchronous control system based on a ball head camera provided in an embodiment of the present invention. The remote synchronous control system based on a ball head camera includes:
[0065] A head mounted display for presenting virtual images to a user wearing the device;
[0066] An inertial measurement unit, which is disposed in the head mounted display and is used to collect motion state information of the head mounted display;
[0067] A control unit connected to the inertial measurement unit and configured to generate an action control instruction according to the action state information;
[0068] A ball head camera, which is used to collect images of the surrounding environment;
[0069] A driving unit connected to the control unit, and configured to drive the ball head camera to perform corresponding synchronous actions according to the action control instruction, so that the ball head camera can collect images of the surrounding environment during the synchronous actions;
[0070] The control unit is also used to convert the surrounding environment image into an image data stream, and send the image data stream back to the head mounted display;
[0071] The head mounted display is also used to convert the image data stream into an image display signal, so as to display the virtual image consistent with the surrounding environment image;
[0072] A sight line determination unit, which is disposed in the head mounted display and is used to collect the viewing sight line information of the user;
[0073] The display adjustment unit is arranged in the head mounted display and is used for adjusting the display state of the virtual image on the head mounted display according to the viewing line of sight information.
[0074] The beneficial effects of the above technical solution are as follows: the remote synchronous control system based on the ball head camera uses inertial measurement to collect the action state information of the head mounted display, and uses this action control instruction; and uses the action control instruction to drive the ball head camera to perform corresponding synchronous actions, so that the ball head camera collects the surrounding environment image during the synchronous action; and then converts the surrounding environment image into an image data stream and sends it back to the head mounted display, so that the head mounted display displays a virtual image consistent with the surrounding environment image to the user; the user's viewing line of sight information is also collected through the line of sight determination unit, and the display state of the virtual image on the head mounted display is adjusted according to the viewing line of sight information through the display adjustment unit. The above system controls the ball head camera and the head mounted display in a linkage manner, and takes the head movement of the user when wearing the head mounted display as a reference to drive the head mounted display to move, and linkage controls the ball head camera to synchronously change its own camera state, so that the user can realize remote control of the ball head camera, and at the same time, the image captured by the ball head camera is presented to the user in real time, thereby improving the real-time and controllability of the image display of the head mounted display, and improving the user experience of the head mounted display.
[0075] Preferably, the inertial measurement unit collects the action state information of the head mounted display specifically including:
[0076] The inertial measurement unit includes an attitude angle measurement subunit and an accelerometer unit;
[0077] The attitude angle measurement subunit is used to detect the action attitude angle information of the head mounted display when the user wears the head mounted display to perform an action; wherein the action attitude angle information includes the pitch angle and the heading angle of the head mounted display during the action;
[0078] The accelerometer unit is used to detect the motion acceleration information of the head mounted display when the user wears the head mounted display to perform actions.
[0079] The beneficial effects of the above technical solution are as follows: in actual work, the inertial measurement unit may include a three-axis acceleration sensor and an acceleration sensor, the three-axis acceleration sensor is used to detect the motion posture angle information of the head-mounted display when the user wears the head-mounted display to perform actions; the acceleration sensor is used to detect the movement acceleration of the head-mounted display when the user wears the head-mounted display to perform actions, so that the motion posture and motion angular velocity of the head-mounted display can be comprehensively detected when the user wears the head-mounted display to perform actions, thereby providing a reliable basis for the subsequent adjustment of the shooting action of the ball head camera.
[0080] Preferably, the control unit generates the action control instruction according to the action state information, specifically comprising:
[0081] The control unit determines the angle change range and angle change speed of the pitch angle and the heading angle of the head mounted display during the action process according to the action posture angle information;
[0082] Then, according to the angle change range and the angle change speed, a first action control instruction is generated for controlling the ball head camera to adjust the angles in the pitch direction and the heading direction during the synchronous action;
[0083] The control unit generates a second motion control instruction for controlling the ball head camera to adjust the shooting focal length during the synchronous motion according to the motion acceleration information.
[0084] The beneficial effects of the above technical solution are as follows: based on the angle change range and angle change speed of the pitch angle and the heading angle of the head-mounted display during the action process, a first action control instruction is generated, and the first action control instruction is subsequently used to adjust the shooting pitch angle and the shooting heading angle of the ball head camera, so that the ball head camera can change the shooting angle in the pitch direction and the heading direction in synchronization with the current angle action of the head-mounted display, that is, the shooting angle value and the shooting angle change rate are changed synchronously, so that the user can control the shooting field of view of the ball head camera by adjusting his own head movement after wearing the head-mounted display. In addition, based on the action acceleration value of the head-mounted display during the action process, a second action control instruction is generated, and the second action control instruction is subsequently used to adjust the shooting focal length of the ball head camera, that is, when the user wears the head-mounted display, the head-mounted display is driven to perform acceleration or deceleration, and the ball head camera can be controlled to perform shooting operations with a larger or smaller focal length.
[0085] Preferably, before generating the first action control instruction and the second action control instruction, the control unit further includes:
[0086] The control unit performs Kalman filtering on the motion posture angle information and the motion acceleration information, thereby achieving data smoothing processing on the motion posture angle information and the motion acceleration information.
[0087] The beneficial effects of the above technical solution are as follows: the motion posture angle information and motion acceleration information detected by the inertial measurement unit have high data sensitivity, and the noise components of the motion posture angle information and motion acceleration information can be effectively eliminated by performing Kalman filtering on the motion posture angle information and motion acceleration information, thereby avoiding the situation where the shooting angle and shooting focal length of the ball head camera are adjusted jammed due to the presence of noise components, thereby ensuring the smoothness and stability of the adjustment of the shooting angle and shooting focal length of the ball head camera.
[0088] Preferably, the control unit generates, according to the motion acceleration information, a second motion control instruction for controlling the ball head camera to adjust the shooting focal length during the synchronous motion, specifically comprising:
[0089] The control unit performs secondary integration processing on the motion acceleration information to obtain corresponding position information; and then generates the second motion control instruction based on the position information; wherein the second motion control instruction is used to adjust the shooting focal length of the ball head camera, thereby realizing the zooming in or out of the picture shot by the ball head camera.
[0090] The beneficial effect of the above technical solution is: through the above method, the target shooting focal length that the ball head camera needs to be adjusted to can be accurately determined based on the motion acceleration information of the head-mounted display, so as to facilitate the accurate zooming in or out of the shooting picture of the ball head camera.
[0091] Preferably, after the control unit performs secondary integration processing on the motion acceleration information, the corresponding position information obtained specifically includes:
[0092] When obtaining the position information, the AHRS algorithm is used to obtain the attitude information. In the process of using the AHRS algorithm, a gyroscope is required to be used as the only measuring device for the current direction of movement. When using the gyroscope to measure the direction, the gyroscope is controlled to rotate in a plane, and stops at a preset fixed angle and measures the static measurement value of the gyroscope at different fixed angles. The rotation of the gyroscope is controlled by a pulse motor, and a counter is added to the pulse motor. The counter outputs a high pulse signal after the motor rotates a full circle. At the same time, the gyroscope is communicated with through a serial port to obtain the measurement value of the gyroscope and the working voltage value of the gyroscope. Before using the gyroscope to measure the direction, it is determined whether the gyroscope is working normally and whether the working voltage is normal. At the same time, the number of pulses input to the pulse motor between the rising edges of the two high pulse signals output by the counter is collected to determine whether the pulse motor is working normally. According to the working state of the gyroscope and the normal working state of the pulse motor, self-test information is generated and the measurement start of the gyroscope is controlled. The process is as follows:
[0093] Step S1, using the following formula (1), according to the communication status with the gyroscope, determine whether the gyroscope is working normally and whether the working voltage is normal,
[0094]
[0095] In the above formula (1), E gIndicates the working state judgment value of the gyroscope; F indicates the output value of the return information received from the gyroscope. If the return information of the gyroscope is received, F=1, otherwise F=0; {F=1|} indicates that under the condition of F=1, the equation after | is judged to be valid. If it is valid, the function value is 1, if not, the function value is 0. If the condition of F=1 is not satisfied, the function value is directly set to 0; && indicates the logical relationship and; A(d) indicates the frame header data in the dth return information received from the gyroscope; A 0 Represents the general data frame header in the valid return data of the gyroscope; R(d) represents the frame tail data in the dth return information received from the gyroscope; R 0 Indicates the general data frame tail in the valid return data of the gyroscope; U(d) indicates the extraction of data on a specific bit in the dth return information received from the gyroscope, and the data on the specific bit is the current operating voltage of the gyroscope extracted; U 0 Indicates the external power supply voltage provided to the gyroscope; D indicates the total number of return messages received from the gyroscope; It means that if the value of d is substituted from 1 to D into the brackets, if there is one or more d values that make the equation in the brackets valid, then the overall equation is valid, otherwise, the overall equation is not valid;
[0096] If E g =1, indicating that the gyroscope's working status self-test is normal;
[0097] If E g =0, indicating that the gyroscope's working state self-check is abnormal;
[0098] Step S2, using the following formula (2), according to the number of pulses input to the pulse motor between the rising edges of the two high pulse signals output by the counter, to determine whether the pulse motor is working normally.
[0099]
[0100] In the above formula (2), E k represents the working state judgment value of the pulse motor; N represents the number of pulses input to the pulse motor between the rising edges of the two high pulse signals output by the counter; θ 0 It indicates the rotation angle value of the pulse motor when a pulse signal is input; || indicates obtaining the absolute value; Indicates rounding up; Z{} indicates a judgment function. If the formula in the brackets is true, the function value is 1; if the formula in the brackets is not true, the function value is 0;
[0101] If E k =1, indicating that the pulse motor's working state self-test is normal;
[0102] If E k =0, indicating that the pulse motor's working state self-check is abnormal;
[0103] Step S3, using the following formula (3), according to the working state of the gyroscope and the normal working state of the pulse motor, generate self-test information and control the measurement start-up enable of the gyroscope,
[0104]
[0105] In the above formula (3), B represents the self-test information, where the information frame header is A 0 , the frame end is R 0 , self-test information data is E g and E k ; E represents the measurement start enable control value of the gyroscope;
[0106] If E=0, it means that the gyroscope self-check is normal, and the gyroscope direction measurement is started;
[0107] If E=1, it means that the gyroscope self-check is abnormal, then after troubleshooting and repairing the fault according to the self-check information, the above steps S1 to S3 are performed again.
[0108] The beneficial effects of the above technical solution are as follows: using the above formula (1), according to the communication status with the gyroscope, it is judged whether the gyroscope is working normally and whether the working voltage is normal, so as to intelligently and automatically check whether there are problems with the communication of the gyroscope and the collected data, reflecting the intelligence of the system and ensuring the reliability and accuracy of the data collected by the gyroscope; then using the above formula (2), according to the number of pulses input to the pulse motor between the rising edges of the two high pulse signals output by the counter, it is judged whether the pulse motor is working normally, ensuring the accuracy of the motor control and control process, ensuring the reliable control of the motor, and ensuring the accuracy of subsequent control; finally, using the above formula (3), according to the working status of the gyroscope and the normal working status of the pulse motor, self-test information is generated and the measurement start-up of the gyroscope is controlled, so as to ensure the reliability and accuracy of the subsequent direction measurement of the gyroscope to the greatest extent.
[0109] Preferably, the driving unit drives the ball head camera to perform corresponding synchronous actions according to the action control instruction, so that the ball head camera collects images of the surrounding environment during the synchronous action.
[0110] The driving unit includes a first driving sub-unit and a second driving sub-unit;
[0111] The first driving subunit is used to drive the ball head camera to perform a rotation action in a pitch direction and a heading direction synchronized with the head mounted display according to the first action control instruction;
[0112] During the rotation of the ball head camera, the second driving subunit drives the ball head camera to change the shooting focal length according to the second action control instruction.
[0113] The beneficial effects of the above technical solution are as follows: in actual work, the first driving subunit may be but is not limited to a gimbal, which can, according to the first action control instruction, accurately drive the ball head camera to rotate in the pitch direction and the heading direction in real time in synchronization with the head-mounted display; the second driving subunit may be but is not limited to a zoom lens barrel, which is used to change the distance between different lenses inside the ball head camera according to the second action control instruction, thereby ensuring that the ball head camera can accurately increase or decrease the shooting focal length.
[0114] Preferably, the control unit converts the surrounding environment image into an image data stream, and sends the image data stream back to the head mounted display, specifically comprising:
[0115] After the control unit performs image background noise reduction processing on the surrounding environment image, the surrounding environment image is converted into an image data stream matching the image playback format of the head-mounted display, and the image data stream is synchronously sent back to the head-mounted display.
[0116] The head mounted display converts the image data stream into an image display signal, thereby displaying the virtual image consistent with the surrounding environment image, specifically including:
[0117] The head-mounted display generates a left-eye image display signal and a right-eye image display signal according to the image data stream, and inputs the signals to the left-eye display screen and the right-eye display screen of the head-mounted display respectively, so as to display the left-eye image and the right-eye image respectively, so that the head-mounted display displays a virtual three-dimensional image consistent with the surrounding environment image.
[0118] The beneficial effect of the above technical solution is: through the above method, the left eye image display signal and the right eye image display signal are generated based on the surrounding environment image, thereby providing the left eye image and the right eye image for the user's left eye and right eye respectively, making it convenient for the user to watch the virtual three-dimensional image consistent with the surrounding environment image.
[0119] Preferably, the line of sight determination unit collects the viewing line of sight information of the user specifically including:
[0120] The sight line determination unit includes an infrared ray transmitting subunit, an infrared ray receiving subunit and a sight line calculation subunit;
[0121] The infrared ray emitting subunit is used to emit infrared rays to the left eye or the right eye of the user;
[0122] The infrared receiving subunit is used to receive the infrared rays emitted by the infrared emitting subunit and reflected by the left eye or the right eye of the user;
[0123] The sight line calculation subunit is used to determine the real-time sight line direction of the left eye or the right eye of the user according to the infrared rays received by the infrared ray receiving subunit, and use this as the viewing sight line information;
[0124] The display adjustment unit adjusts the display state of the virtual image on the head mounted display according to the viewing line of sight information, specifically including:
[0125] The display adjustment unit is used to adjust the display position of the left-eye image on the left-eye display screen or adjust the display position of the right-eye image on the right-eye display screen according to the viewing line of sight information.
[0126] The beneficial effect of the above technical solution is: through the above method, the display position of the left eye image on the left eye display screen or the display position of the right eye image on the right eye display screen can be adjusted based on the user's left eye and right eye lines of sight, ensuring that the left eye image and the right eye image are always located in the display center area of the left eye display screen and the right eye display screen, thereby improving the user's comfort in viewing virtual three-dimensional images.
[0127] It can be seen from the contents of the above embodiments that the remote synchronous control system based on the ball head camera uses inertial measurement to collect the action state information of the head mounted display, and uses this action control instruction; and uses the action control instruction to drive the ball head camera to perform corresponding synchronous actions, so that the ball head camera collects the surrounding environment image during the synchronous action; and then converts the surrounding environment image into an image data stream and sends it back to the head mounted display, so that the head mounted display displays a virtual image consistent with the surrounding environment image to the user; the user's viewing line of sight information is also collected through the line of sight determination unit, and the display state of the virtual image on the head mounted display is adjusted according to the viewing line of sight information through the display adjustment unit. The above system controls the ball head camera and the head mounted display in a linkage manner, and takes the head movement of the user when wearing the head mounted display as a reference to drive the head mounted display to move, and controls the ball head camera in a linkage manner to synchronously change its own camera state, so that the user can realize remote control of the ball head camera, and at the same time, the image captured by the ball head camera is presented to the user in real time, thereby improving the real-time and controllability of the image display of the head mounted display, and improving the user experience of the head mounted display.
[0128] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.
Claims
1. The remote synchronous control system based on the ball head camera is characterized by: It includes: A head mounted display, which is used to present virtual images to the user wearing it; an inertial measurement unit, which is arranged in the head mounted display and is used to collect motion state information of the head mounted display; A control unit connected to the inertial measurement unit, and configured to generate an action control instruction according to the action state information; A ball head camera, which is used to collect images of the surrounding environment; A driving unit connected to the control unit, and configured to drive the ball head camera to perform corresponding synchronous actions according to the action control instruction, so that the ball head camera can collect images of the surrounding environment during the synchronous actions; The control unit is also used to convert the surrounding environment image into an image data stream, and send the image data stream back to the head mounted display; The head mounted display is further used to convert the image data stream into an image display signal, so as to display the virtual image consistent with the surrounding environment image; A sight line determination unit, which is disposed in the head mounted display and is used to collect the viewing sight line information of the user; A display adjustment unit, which is disposed in the head mounted display and is used to adjust a display state of the virtual image on the head mounted display according to the viewing line of sight information; The control unit generates the action control instruction according to the action state information, specifically comprising: The control unit determines the angle change range and angle change speed of the pitch angle and the heading angle of the head mounted display during the action according to the action posture angle information; Then, according to the angle change range and the angle change speed, a first action control instruction is generated for controlling the ball head camera to adjust the angles in the pitch direction and the heading direction during the synchronous action; The control unit generates, according to the motion acceleration information, a second motion control instruction for controlling the ball head camera to adjust the shooting focal length during the synchronous motion; The control unit generates, according to the motion acceleration information, a second motion control instruction for controlling the ball head camera to adjust the shooting focal length during the synchronous motion, specifically including: The control unit performs secondary integration processing on the motion acceleration information to obtain corresponding position information; and then generates the second motion control instruction according to the position information; wherein the second motion control instruction is used to adjust the shooting focal length of the ball head camera, thereby realizing the zooming in or out of the shooting picture of the ball head camera; The control unit performs secondary integration processing on the motion acceleration information to obtain corresponding position information, which specifically includes: When obtaining the position information, the AHRS algorithm is used to obtain the attitude information. In the process of using the AHRS algorithm, a gyroscope is used as the only measuring device for the current direction of motion. When the gyroscope is used to measure the direction, the gyroscope is controlled to rotate in a plane, and stops at a preset fixed angle and measures the static measurement value of the gyroscope at different fixed angles. The rotation of the gyroscope is controlled by a pulse motor, and a counter is added to the pulse motor. The counter outputs a high pulse signal after the motor rotates a full circle, and communicates with the gyroscope through a serial port to obtain the measurement value of the gyroscope and the working voltage value of the gyroscope. Before using the gyroscope to measure the direction, it is determined whether the gyroscope is working normally and whether the working voltage is normal. At the same time, the number of pulses input to the pulse motor between the rising edges of the two high pulse signals output by the counter is collected to determine whether the pulse motor is working normally. Self-test information is generated according to the working state of the gyroscope and the normal working state of the pulse motor, and the measurement start-up of the gyroscope is controlled. The process is as follows: Step S1, using the following formula (1), according to the communication status with the gyroscope, determine whether the gyroscope is working normally and whether the working voltage is normal, In the above formula (1), E g represents the working state judgment value of the gyroscope; F represents the output value of the return information received from the gyroscope. If the return information of the gyroscope is received, F=1, otherwise F=0; {F=1|} represents that under the condition of satisfying F=1, the equation behind | is judged whether it is established. If it is established, the function value is 1, if not, the function value is 0, and if the condition of F=1 is not satisfied, the function value is directly set to 0; && represents the logical relationship and; A(d) represents the frame header data in the dth return information received from the gyroscope; A0 represents the general data frame header in the effective return data of the gyroscope; R(d) represents the frame tail data in the dth return information received from the gyroscope; R0 represents the general data frame tail in the effective return data of the gyroscope; U(d) represents the data on a specific bit of the dth return information received from the gyroscope, and the data on the specific bit is the extracted current working voltage of the gyroscope; U0 represents the external power supply voltage provided to the gyroscope; D represents the total number of return information received from the gyroscope; It means that if the value of d is substituted from 1 to D into the brackets, if there is one or more d values that make the equation in the brackets valid, then the overall equation is valid, otherwise, the overall equation is not valid; If E g =1, indicating that the gyroscope's working state self-test is normal; If E g =0, indicating that the gyroscope's working state self-check is abnormal; Step S2, using the following formula (2), according to the number of pulses input to the pulse motor between the rising edges of two high pulse signals output by the counter, to determine whether the pulse motor is working normally, In the above formula (2), E k represents the working state judgment value of the pulse motor; N represents the number of pulses input to the pulse motor between the rising edges of two high pulse signals output by the counter; θ0 represents the rotation angle value of a pulse signal input to the pulse motor; It means to find the absolute value; Indicates rounding up; Represents a judgment function. If the formula in the brackets is true, the function value is 1. If the formula in the brackets is not true, the function value is 0. If E k =1, indicating that the pulse motor's working state self-test is normal; If E k =0, indicating that the pulse motor's working state self-check is abnormal; Step S3, using the following formula (3), according to the working state of the gyroscope and the normal working state of the pulse motor, generating self-test information and controlling the measurement start-up enable of the gyroscope, In the above formula (3), B represents the self-test information, where the information frame header is A0, the frame tail is R0, and the self-test information data is E g and E k ; E represents the measurement start enable control value of the gyroscope; If E=0, it means that the gyroscope self-check is normal, and the direction measurement of the gyroscope is started; If E=1, it means that the gyroscope self-check is abnormal, then after troubleshooting and repairing according to the self-check information, the above steps S1 to S3 are performed again.
2. The remote synchronous control system based on the ball head camera as claimed in claim 1, characterized in that: The inertial measurement unit collects the action state information of the head mounted display specifically including: The inertial measurement unit includes an attitude angle measurement subunit and an accelerometer unit; The attitude angle measurement subunit is used to detect the action attitude angle information of the head mounted display when the user wears the head mounted display to perform an action; wherein the action attitude angle information includes the pitch angle and the heading angle of the head mounted display during the action; The accelerometer unit is used to detect the motion acceleration information of the head mounted display when the user wears the head mounted display and performs an action.
3. The remote synchronous control system based on the ball head camera as claimed in claim 1, characterized in that: Before generating the first action control instruction and the second action control instruction, the control unit also includes The control unit performs Kalman filtering on the motion posture angle information and the motion acceleration information, thereby achieving data smoothing processing on the motion posture angle information and the motion acceleration information.
4. The remote synchronous control system based on a ball head camera as claimed in claim 1, characterized in that: The driving unit drives the ball head camera to perform corresponding synchronous actions according to the action control instruction, so that the ball head camera collects images of the surrounding environment during the synchronous action. The driving unit comprises a first driving sub-unit and a second driving sub-unit; The first driving subunit is used to drive the ball head camera to perform a rotation action in a pitch direction and a heading direction synchronized with the head mounted display according to the first action control instruction; During the rotation of the ball head camera, the second driving subunit drives the ball head camera to change the shooting focal length according to the second action control instruction.
5. The remote synchronous control system based on the ball head camera as claimed in claim 4, characterized in that: The control unit converts the surrounding environment image into an image data stream, and sends the image data stream back to the head mounted display, specifically comprising: After the control unit performs image background noise reduction processing on the surrounding environment image, the control unit converts the surrounding environment image into an image data stream that matches the image playback format of the head-mounted display, and synchronously sends the image data stream back to the head-mounted display; The head mounted display converts the image data stream into an image display signal, thereby displaying the virtual image consistent with the surrounding environment image, specifically including: The head-mounted display generates a left-eye image display signal and a right-eye image display signal according to the image data stream, and inputs the signals to the left-eye display screen and the right-eye display screen of the head-mounted display, respectively, so as to display the left-eye image and the right-eye image, respectively, so that the head-mounted display displays a virtual three-dimensional image consistent with the surrounding environment image.
6. The remote synchronous control system based on the ball head camera as claimed in claim 5, characterized in that: The sight line determination unit collecting the viewing sight line information of the user specifically includes: The sight line determination unit includes an infrared ray transmitting subunit, an infrared ray receiving subunit and a sight line calculation subunit; The infrared emitting subunit is used to emit infrared rays to the left eye or the right eye of the user; The infrared receiving subunit is used to receive infrared rays emitted by the infrared emitting subunit and reflected by the left eye or the right eye of the user; The sight line calculation subunit is used to determine the real-time sight line direction of the left eye or the right eye of the user according to the infrared rays received by the infrared receiving subunit, and use this as the viewing sight line information; The display adjustment unit adjusts the display state of the virtual image on the head mounted display according to the viewing line of sight information, specifically comprising: The display adjustment unit is used to adjust the display position of the left-eye image on the left-eye display screen or adjust the display position of the right-eye image on the right-eye display screen according to the viewing line of sight information.
Citation Information
Patent Citations
Operation pointer indication control device based on human eye image and sight tracking
CN104090659A
Virtual augmented reality system for realizing remote control based on head-mounted device
CN106249847A
Excavator remote control method based on VR equipment
CN107119739A
Real-time control system and method of underwater robot video camera holder
CN107968915A
Cockpit system based on mixed reality, and scene construction method
CN112150885A
