A large-screen touch interactive conference room multimedia terminal based on machine vision
By using machine vision technology and convolutional neural network algorithm to identify fingers or touch pens in conference rooms, and combining geometric measurement technology for positioning, the problems of high price and poor adaptability of large-screen touch functions in the existing technology are solved, and high Integration and precise positioning of multimedia terminals.
Patent Information
- Application Number
- CN202410461716.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-17
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-04-17
AI Technical Summary
The touch function of the existing conference large screen is expensive and has low integration. It cannot be adapted to large screens of different sizes, so it is difficult to use and install and calibration.
Design a large-screen touch interactive conference room multimedia terminal based on machine vision, using two 180° fisheye high-definition cameras for image processing, identify fingers or tweak pens through convolutional neural network algorithm, and position them in combination with geometric measurement technology to achieve accurate 2D coordinate transmission simplifies the calibration process.
It realizes a high-integration equipment design, simplifies installation and calibration operations, reduces costs, adapts to large screens of different sizes, and uses algorithms to shield external light interference, realizing accurate positioning of fingers and touch pens.
Smart Images

Figure CN118276711B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of camera image processing algorithms, and in particular relates to a large-screen touch interactive conference room multimedia terminal based on machine vision. Background Art
[0002] The human-computer interaction experience of conference terminals based on remote controls is very unfriendly. Since the human-computer interaction methods of Android and Windows systems are designed based on click logic, if you use the remote control for human-computer interaction, you need to press a lot of buttons to achieve an effective operation of touch screen click, which is very inconvenient.
[0003] Some terminal manufacturers integrate the air mouse function into the remote control. Through the gyroscope sensor, users can control the movement of the mouse by waving the remote control. However, this method is not a very good experience, and there are problems such as gyroscope calibration, mouse drift, and remote control loss.
[0004] Some large conference screens have touch functions, but they are relatively expensive.
[0005] There are also after-installed large-screen touch solutions on the market, such as infrared touch frames and laser touch frames. On the one hand, such solutions are relatively expensive and have low integration. The same product cannot be adapted to large screens of different sizes, and it is difficult to use, install and calibrate.
[0006] Based on this, the present invention designs a large-screen touch interactive conference room multimedia terminal based on machine vision to solve the above problems. Summary of the invention
[0007] The purpose of the present invention is to solve the problem that some large conference screens have touch functions but are relatively expensive. There are also post-installed large-screen touch solutions on the market, such as infrared touch frames and laser touch frames. On the one hand, such solutions are relatively expensive and have low integration. The same product cannot be adapted to large screens of different sizes, and the installation and calibration are difficult. A large-screen touch interactive conference room multimedia terminal based on machine vision is proposed.
[0008] In order to achieve the above object, the present invention adopts the following technical solutions:
[0009] A large-screen touch interactive conference room multimedia terminal based on machine vision, including conference audio and video terminal equipment, the conference audio and video terminal equipment includes two 180° fisheye high-definition cameras, a conference high-definition camera, an array microphone, a device bracket, a hidden speaker and an interface module, the two 180° fisheye high-definition cameras are used to capture images of fingers touching the screen, the conference high-definition camera is used for video conferencing, the array microphone is used to pick up sound in the conference room environment, the device bracket is used to fix the device on the top of the large screen, the hidden speaker is used to play audio in the meeting, the interface module contains HDMI output / input, USB interface, power input port and RJ45 network interface, the conference audio and video terminal equipment is connected to the large screen through an HDMI cable, so as to transmit video and audio signals to the large screen;
[0010] The two 180° fisheye high-definition cameras capture finger or touch pen movements in the large screen area. The images captured by the two 180° fisheye high-definition cameras are first processed in the video DSP chip, and the background and fingers or touch pen are identified through noise elimination and convolutional neural network algorithms. The finger or touch pen is extracted from the background, and two-point positioning is performed according to the two 180° fisheye high-definition cameras to give the relative two-dimensional x and y axis coordinates of the finger or touch pen on the screen, and the coordinates are sent to the MCU. The MCU transmits the processed coordinates and click information to the main chip, and the main chip gives corresponding actions on the large screen after processing.
[0011] As a further description of the above technical solution:
[0012] The two 180° fisheye high-definition cameras perform two-point positioning to provide the relative two-dimensional x and y axis coordinates of the finger or touch pen on the screen. Based on the two-corner positioning technology and the two-side positioning technology in the geometric measurement technology, the position coordinates of the finger or touch pen on the large screen are obtained by measuring the distance and angle.
[0013] As a further description of the above technical solution:
[0014] In the two-side positioning method, the coordinates of the two 180° fisheye high-definition cameras as positioning beacons are known, namely A (x1, y1, z1) and B (x2, y2, z2), and the finger or touch pen C (x c ,y c ,y c ) are obtained by using WiFi signal strength ranging technology, and the following equations are obtained:
[0015]
[0016] The positioning target is on the two-dimensional plane of the large screen, and the inherent Z1=Z2, substituting into the above formula and making a difference again, we can get:
[0017]
[0018] By converting the above formula into a matrix and combining it with the least squares method, the relative two-dimensional x and y axis coordinates of the finger or touch pen can be obtained.
[0019] As a further description of the above technical solution:
[0020] The conference audio and video terminal device needs to run a calibration program when it is used for the first time. When the conference audio and video terminal device is securely fixed on the top of the large screen, the calibration program is started. During calibration, the large screen will display a cross cursor at the four corners. At this time, use your finger or touch pen to click the center of the cross cursor in turn to complete the calibration. After the calibration is completed, it can be used normally. If the relative position of the conference audio and video terminal device and the large screen has not changed, there is no need to calibrate again.
[0021] As a further description of the above technical solution:
[0022] The main chip of the conference audio and video terminal device can use an ARM-based main chip and run the Android system according to needs, and can also use a main chip based on the X86 architecture and run Windows or use other main chips to run the system supported by the chip.
[0023] As a further description of the above technical solution:
[0024] The data stream processing process of the two 180° fisheye high-definition cameras includes:
[0025] The DSP chip first calibrates the 180° fisheye HD camera lens distortion to make the image with distorted edges straight;
[0026] Then the part of the field of view that is not relevant to touch is cropped, and then the grayscale image is obtained by weighted average of the RGB image of each pixel. The static background in the image is eliminated by weighted subtraction of the previous and next frames.
[0027] In order to eliminate the influence of the change of the large screen display content itself on the recognition, the DSP chip will also input the image display signal of the main chip, which will be eliminated as the background after deformation and grayscale fitting;
[0028] The subtracted image above will also filter out moving objects in the image. By adjusting the weight data of the previous and next frames, the sensitivity of change recognition can be adjusted.
[0029] Then, the edges of moving objects are extracted through the first-order derivative edge operator and the Canny edge detection algorithm based on the Sobel operator;
[0030] The extracted image is fed into the trained convolutional neural network, which calculates the actual position of the finger on the screen based on the previously trained parameters.
[0031] The training method is to generate coordinates on the screen and then let people click with their hands or a stylus. At this time, the clicking method is trained in combination with different lighting conditions.
[0032] As a further description of the above technical solution:
[0033] The convolutional layer and the connection layer train internal parameters according to the input edge detection image and the output actual position of the finger or touch pen on the screen during learning until fitting.
[0034] As a further description of the above technical solution:
[0035] The convolutional neural network output layer generates coordinate parameters of multiple fingers, such as finger 1 (x1, y2) and finger 2 (x2, y2), and communicates with the microcontroller through the serial port.
[0036] As a further description of the above technical solution:
[0037] The single chip microcomputer converts the received signal into a signal conforming to the HID external device protocol, communicates with the main chip, and controls the display content on the main chip system based on the signal.
[0038] As a further description of the above technical solution:
[0039] The Canny edge detection algorithm uses a gradient amplitude histogram to perform concavity analysis, and automatically selects a double threshold value according to the characteristics of the image. The double threshold value selection step includes:
[0040] Calculate the 256-level gradient magnitude histogram s(i) of the image and the non-zero starting point (i) of the histogram s(i). start ,s(i start )) and non-zero endpoint (i end ,s(i end )), and save these two points;
[0041] Calculate the gradient amplitude in the histogram by i start to end The slope of is calculated by the following formula:
[0042]
[0043] Where p(i) represents the slope and i represents the gradient amplitude;
[0044] Find the maximum value of p(i) and the corresponding amplitude i peak , (i peak ,s(i peak )) is a convex point of the gradient histogram. If i peak <i end , then let i start =i peak , go to formula Perform secondary calculations;
[0045] In summary, we obtain N convex points (i n1 ,s(i n1 ))、(i n2 ,s(i n2 ))、…、(i nx ,s(i nx )), connect the non-zero starting points (i start ,s(i stert )) and non-zero endpoint (i end ,s(i end )) and the N convex points can obtain the minimum convex polygon of an envelope histogram s(i)
[0046] Calculate the concave residual k(i) of the histogram, When the concave residual takes the maximum value, the corresponding gradient amplitude is the high threshold G s , the lower threshold is G1 = 0.4*G s .
[0047] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:
[0048] 1. In the present invention, the conference audio and video terminal equipment includes two 180° fisheye high-definition cameras, a high-definition camera for conference, an array microphone, an equipment bracket, a hidden speaker and an interface module with high integration, and the installation and calibration operations are simple and fast. The convolution layer and the connection layer train the internal parameters according to the input edge detection image and the output actual position of the finger or touch pen on the screen during learning until fitting. The convolutional neural network output layer will generate coordinate parameters of multiple fingers, such as finger 1 (x1, y2) and finger 2 (x2, y2), and communicate with the single-chip microcomputer through the serial port. The single-chip microcomputer converts the received signal into a signal that conforms to the HID external device protocol, communicates with the main chip, and controls the display content on the main chip system based on this.
[0049] 2. In the present invention, the two 180° fisheye high-definition cameras capture the finger or touch pen movements in the large screen area. The images captured by the two 180° fisheye high-definition cameras are first processed in the video DSP chip, and the background and the finger or touch pen are identified through noise elimination and convolutional neural network algorithm. The finger or touch pen is extracted from the background, and two-point positioning is performed according to the two 180° fisheye high-definition cameras to give the relative two-dimensional x and y axis coordinates of the finger or touch pen on the screen, and the coordinates are sent to the MCU, and the MCU will process the processed coordinates and point The tap information is transmitted to the main chip, which processes it and gives a corresponding action on the large screen. The two 180° fisheye high-definition cameras perform two-point positioning to give the relative two-dimensional x and y axis coordinates of the finger or touch pen on the screen. Based on the two-corner positioning technology and the two-side positioning technology in the geometric measurement technology, the position coordinates of the finger or touch pen on the large screen are obtained by measuring the distance and angle respectively. The algorithm can effectively shield the interference of external light, realize the precise positioning of the finger and touch pen, and realize basic communication. Then the two 180° fisheye high-definition cameras collect signals for positioning. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] Figure 1 A schematic diagram of the processing flow of a large-screen touch interactive conference room multimedia terminal based on machine vision proposed by the present invention;
[0051] Figure 2 This is a schematic diagram of data stream processing of a 180° fisheye high-definition camera in a large-screen touch interactive conference room multimedia terminal based on machine vision proposed by the present invention;
[0052] Figure 3 A schematic diagram of the calibration of conference audio and video terminal equipment in a large-screen touch interactive conference room multimedia terminal based on machine vision proposed by the present invention;
[0053] Figure 4 This is a schematic diagram of the data stream processing process of a 180° fisheye high-definition camera in a large-screen touch interactive conference room multimedia terminal based on machine vision proposed by the present invention. DETAILED DESCRIPTION
[0054] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0055] Please see attached Figure 1 -Attached Figure 4The present invention provides a technical solution: a large-screen touch interactive conference room multimedia terminal based on machine vision, including a conference audio and video terminal device, the conference audio and video terminal device includes two 180° fisheye high-definition cameras, a conference high-definition camera, an array microphone, a device bracket, a hidden speaker and an interface module, the two 180° fisheye high-definition cameras are used to capture images of fingers touching the screen, the conference high-definition camera is used for video conferencing, the array microphone is used to pick up sound in the conference room environment, the device bracket is used to fix the device on the top of the large screen, the hidden speaker is used to play audio in the meeting, the interface module contains HDMI output / input, USB interface, power input port and RJ45 network interface, the conference audio and video terminal device is connected to the large screen through an HDMI cable to transmit video and audio signals to the large screen;
[0056] The two 180° fisheye high-definition cameras capture finger or touch pen movements in the large screen area. The images captured by the two 180° fisheye high-definition cameras are first processed in the video DSP chip, and the background and fingers or touch pen are identified through noise elimination and convolutional neural network algorithms. The finger or touch pen is extracted from the background, and two-point positioning is performed according to the two 180° fisheye high-definition cameras to give the relative two-dimensional x and y axis coordinates of the finger or touch pen on the screen, and the coordinates are sent to the MCU. The MCU transmits the processed coordinates and click information to the main chip, and the main chip gives corresponding actions on the large screen after processing.
[0057] Specifically, the two 180° fisheye high-definition cameras perform two-point positioning to provide the relative two-dimensional x and y axis coordinates of the finger or touch pen on the screen. Based on the two-corner positioning technology and the two-side positioning technology in the geometric measurement technology, the position coordinates of the finger or touch pen on the large screen are obtained by measuring the distance and angle.
[0058] Specifically, in the two-side positioning method, the coordinates of the two 180° fisheye high-definition cameras as positioning beacons are known, namely A (x1, y1, z1) and B (x2, y2, z2), and the coordinates of the finger or touch pen C (x c ,y c ,y c ) are obtained by using WiFi signal strength ranging technology, and the following equations are obtained:
[0059]
[0060] The positioning target is on the two-dimensional plane of the large screen, and the inherent Z1=Z2, substituting into the above formula and making a difference again, we can get:
[0061]
[0062] By converting the above formula into a matrix and combining it with the least squares method, the relative two-dimensional x and y axis coordinates of the finger or touch pen can be obtained.
[0063] Specifically, the conference audio and video terminal device needs to run a calibration program when it is used for the first time. When the conference audio and video terminal device is securely fixed on the top of the large screen, the calibration program is started. During calibration, the large screen will display a cross cursor at the four corners. At this time, use your finger or touch pen to click the center of the cross cursor in turn to complete the calibration. After the calibration is completed, it can be used normally. If the relative position of the conference audio and video terminal device and the large screen has not changed, there is no need to calibrate again.
[0064] Specifically, the main chip of the conference audio and video terminal device can use an ARM-based main chip and run the Android system according to needs, or can use a main chip based on the X86 architecture and run Windows or use other main chips to run the system supported by the chip.
[0065] Specifically, the data stream processing process of the two 180° fisheye high-definition cameras includes:
[0066] The DSP chip first calibrates the 180° fisheye HD camera lens distortion to make the image with distorted edges straight;
[0067] Then the part of the field of view that is not relevant to touch is cropped, and then the grayscale image is obtained by weighted average of the RGB image of each pixel. The static background in the image is eliminated by weighted subtraction of the previous and next frames.
[0068] In order to eliminate the influence of the change of the large screen display content itself on the recognition, the DSP chip will also input the image display signal of the main chip, which will be eliminated as the background after deformation and grayscale fitting;
[0069] The subtracted image above will also filter out moving objects in the image. By adjusting the weight data of the previous and next frames, the sensitivity of change recognition can be adjusted.
[0070] Then, the edges of moving objects are extracted through the first-order derivative edge operator and the Canny edge detection algorithm based on the Sobel operator;
[0071] The extracted image is fed into the trained convolutional neural network, which calculates the actual position of the finger on the screen based on the previously trained parameters.
[0072] The training method is to generate coordinates on the screen and then let people click with their hands or a stylus. At this time, the clicking method is trained in combination with different lighting conditions.
[0073] Specifically, during learning, the convolutional layer and the connection layer train internal parameters according to the input edge detection image and the output actual position of the finger or touch pen on the screen until fitting.
[0074] Specifically, the convolutional neural network output layer generates coordinate parameters of multiple fingers, such as finger 1 (x1, y2) and finger 2 (x2, y2), and communicates with the microcontroller through the serial port.
[0075] Specifically, the single chip microcomputer converts the received signal into a signal that complies with the HID external device protocol, communicates with the main chip, and controls the display content on the main chip system based on the signal.
[0076] Specifically, the Canny edge detection algorithm uses a gradient amplitude histogram to perform concavity analysis, and automatically selects a double threshold value according to the characteristics of the image. The double threshold value selection step includes:
[0077] Calculate the 256-level gradient magnitude histogram s(i) of the image and the non-zero starting point (i) of the histogram s(i). start ,s(i start )) and non-zero endpoint (i end ,s(i end )), and save these two points;
[0078] Calculate the gradient amplitude in the histogram by i start to end The slope of is calculated by the following formula:
[0079]
[0080] Where p(i) represents the slope and i represents the gradient amplitude;
[0081] Find the maximum value of p(i) and the corresponding amplitude i peak , (i peak ,s(i peak )) is a convex point of the gradient histogram. If i peak <i end , then let i start =i peak , go to formula Perform secondary calculations;
[0082] In summary, we obtain N convex points (i n1 ,s(i n1 ))、(i n2 ,s(i n2 ))、…、(i nx ,s(i nx )), connect the non-zero starting points (i start ,s(istart )) and non-zero endpoint (i end ,s(i end )) and the N convex points can obtain the minimum convex polygon of an envelope histogram s(i)
[0083] Calculate the concave residual k(i) of the histogram, When the concave residual takes the maximum value, the corresponding gradient amplitude is the high threshold G s , the lower threshold is G1 = 0.4*G s .
[0084] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.
Claims
1. A large-screen touch interactive conference room multimedia terminal based on machine vision, including conference audio and video terminal equipment, characterized in that: The conference audio and video terminal device includes two 180° fisheye high-definition cameras, a conference high-definition camera, an array microphone, a device bracket, a hidden speaker and an interface module. The two 180° fisheye high-definition cameras are used to capture images of fingers touching the screen, the conference high-definition camera is used for video conferencing, the array microphone is used to pick up sound in the conference room environment, the device bracket is used to fix the device on the top of the large screen, the hidden speaker is used to play audio during the meeting, and the interface module contains HDMI output / input, USB interface, power input port and RJ45 network interface. The conference audio and video terminal device is connected to the large screen via an HDMI cable to transmit video and audio signals to the large screen; The two 180° fisheye high-definition cameras capture the finger or touch pen movements in the large screen area. The images captured by the two 180° fisheye high-definition cameras are first processed in the video DSP chip. The background and the finger or touch pen are identified through noise elimination and convolutional neural network algorithms. The finger or touch pen is extracted from the background, and two-point positioning is performed according to the two 180° fisheye high-definition cameras to give the relative two-dimensional x and y axis coordinates of the finger or touch pen on the screen, and the coordinates are sent to the MCU. The MCU transmits the processed coordinates and click information to the main chip, and the main chip gives corresponding actions on the large screen after processing; The two 180° fisheye high-definition cameras perform two-point positioning to provide the relative two-dimensional x and y axis coordinates of the finger or touch pen on the screen. Based on the two-corner positioning technology and the two-side positioning technology in the geometric measurement technology, the position coordinates of the finger or touch pen on the large screen are obtained by measuring the distance and angle respectively; In the two-side positioning method, the coordinates of the two 180° fisheye high-definition cameras as positioning beacons are known, namely A (x1, y1, z1) and B (x2, y2, z2), and the coordinates of the finger or touch pen C (x C ,y C ,y C ) are obtained by using WiFi signal strength ranging technology, and the following equations are obtained: The positioning target is on the two-dimensional plane of the large screen, and the inherent Z1=Z2, substituting into the above formula and making a difference again, we can get: Converting the above formula into a matrix and combining it with the least squares method can obtain the relative two-dimensional x and y axis coordinates of the finger or touch pen; The data stream processing process of the two 180° fisheye high-definition cameras includes: The DSP chip first calibrates the 180° fisheye HD camera lens distortion to make the image with distorted edges straight; Then, the part of the field of view that is irrelevant to the touch is cropped, and then the grayscale image is obtained by weighted averaging the RGB image of each pixel, and the static background in the image is eliminated by weighted subtraction of the previous and next frames. In order to eliminate the influence of the change of the large-screen display content itself on the recognition, the DSP chip will also input the image display signal of the main chip, which will be eliminated as the background after deformation and grayscale fitting. The image after the above subtraction will also filter out the moving objects in the image, and the sensitivity to the change recognition can be adjusted by adjusting the weight data of the previous and next frames. Then, the edges of moving objects are extracted through the first-order derivative edge operator and the Canny edge detection algorithm based on the Sobel operator; The extracted image is fed into a trained convolutional neural network, which calculates the actual position of the finger on the screen based on previously trained parameters. The training method is to generate coordinates on the screen and then let people click with their hands or stylus. At this time, the click method is trained in different lighting conditions; During learning, the convolutional layer and the connection layer train the internal parameters according to the input edge detection image and the output actual position of the finger or stylus on the screen until the fit is achieved; The output layer of the convolutional neural network generates coordinate parameters of multiple fingers and communicates with the microcontroller through the serial port; The single chip microcomputer converts the received signal into a signal conforming to the HID external device protocol, communicates with the main chip, and controls the display content on the main chip system based on the signal; The Canny edge detection algorithm uses a gradient amplitude histogram to perform concavity analysis, and automatically selects a double threshold value according to the characteristics of the image. The double threshold value selection step includes: Calculate the 256-level gradient magnitude histogram s(i) of the image and the non-zero starting point (i) of the histogram s(i). start ,s(i start )) and non-zero endpoint (i end ,s(i end )), and save these two points; Calculate the gradient amplitude in the histogram by i start to end The slope of is calculated by the following formula: Where p(i) represents the slope and i represents the gradient amplitude; Find the maximum value of p(i) and the corresponding amplitude i peak , (i peak ,s(i peak )) is a convex point of the gradient histogram. If i peak end , then let i start =i peak , go to formula Perform secondary calculations; In summary, we obtain N convex points (i n1 ,s(i n1 ))、(i n2 ,s(i n2 ))、…、(i nx ,s(i nx )), connect the non-zero starting points (i start ,s(i stert )) and non-zero endpoint (i end ,s(i end )) and the N convex points can obtain the minimum convex polygon of an envelope histogram s(i) Calculate the concave residual k(i) of the histogram, When the concave residual takes the maximum value, the corresponding gradient amplitude is the high threshold G S , the lower threshold is G1=0.4*G S .
2. According to the machine vision-based large-screen touch interactive conference room multimedia terminal of claim 1, it is characterized in that: The conference audio and video terminal device needs to run a calibration program when it is used for the first time. When the conference audio and video terminal device is securely fixed on the top of the large screen, the calibration program is started. During calibration, the large screen will display a cross cursor at the four corners. At this time, use your finger or touch pen to click the center of the cross cursor in turn to complete the calibration. After the calibration is completed, it can be used normally. If the relative position of the conference audio and video terminal device and the large screen has not changed, there is no need to calibrate again.
3. According to the machine vision-based large-screen touch interactive conference room multimedia terminal of claim 2, it is characterized in that: The main chip of the conference audio and video terminal device can use an ARM-based main chip and run the Android system according to needs, and can also use a main chip based on the X86 architecture and run Windows or use other main chips to run the system supported by the chip.
