Interaction device based on infrared image object recognition

Abstract

The invention discloses an interaction device based on infrared image object recognition, and the device comprises an infrared camera module which is used for obtaining an infrared image of an interaction experience module in real time, and the infrared image comprises an interaction desktop, an interaction prop, and a gesture posture image; the detection main server comprises a prop recognition module, a coordinate simulation module and a gesture recognition module, the prop recognition module recognizes the relative position information of the interactive prop in the infrared image, and the coordinate simulation module obtains projection area coordinates of the interactive prop based on the relative position information; the gesture posture recognition module recognizes the gesture posture image; an interactive experience server; and the fusion server is used for processing the corresponding interactive image data and projecting the interactive image data to the interactive experience module. According to the invention, the problems of single interaction mode and unstable operation caused by difficult type identification of multi-type interaction objects and low-cost interaction display design of the existing interaction device can be solved.

Description

Technical Field

[0001] The invention relates to the technical field of exhibitions, in particular to an interactive device based on infrared image object recognition. Background Art

[0002] Immersive experience spaces are often set up in various theme halls, museums, and planning halls. The interactive methods of such experience spaces are generally composed of immersive films and interactive special effects. Among them, the interactive mechanism is mostly to detect the location of the audience in the space through radar and other technologies, and associate the interactive devices at the corresponding positions to interact with the game. The associated interactive design of multiple categories of objects in the immersive space requires that the interactive objects be equipped with interactive modules such as light emission and audio playback, which makes the cost high. Generally, specific object categories can be trained and identified using Al technology. However, this type of object detection technology is very sensitive to lighting and ambient light, and different light brightness has a great impact on the detection results. Especially when there are constantly changing projection images in the detection scene, the accuracy of this vision-based object detection becomes extremely low, making it difficult to apply in practice. Summary of the invention

[0003] The purpose of the present invention is to provide an interactive device based on infrared image object recognition in order to solve the problem that existing interactive devices are difficult to identify the types of multiple categories of interactive objects and have low-cost interactive display design, resulting in a single interactive mode and unstable operation.

[0004] In order to achieve the above object, the present invention adopts the following technical solution: an interactive device based on infrared image object recognition, comprising:

[0005] An infrared camera module, which is used to obtain infrared images of an interactive experience module in real time. The interactive experience module includes an interactive desktop and interactive props. The infrared images include an interactive desktop image, an interactive prop image, and a gesture image of a person.

[0006] The detection main server includes a prop recognition module, a coordinate simulation module, a gesture recognition module and a data transmission module. The prop recognition module recognizes the relative position information of the interactive desktop and the interactive props on the acquired infrared image. The coordinate simulation module puts the pre-entered interactive prop information model into the corresponding position of the three-dimensional coordinate system with the interactive desktop model based on the relative position information, and obtains the projection area coordinates of the interactive prop. The gesture recognition module recognizes the gesture image to form gesture information.

[0007] Interactive experience server, which is used for interactive program control and operation;

[0008] A fusion server processes the corresponding interactive image data based on the projection area coordinates and the gesture posture information, and projects the data onto the interactive experience module through a projection interaction module.

[0009] As a further description of the above technical solution:

[0010] The detection main server also includes a scanning and system building module. The scanning and system building module scans the appearance of the interactive desktop and the projection interactive module based on the infrared camera module, and builds the three-dimensional coordinate system with models of the two.

[0011] As a further description of the above technical solution:

[0012] The detection main server also includes a prop entry module, through which the interactive prop information is entered for use in the coordinate simulation module to obtain the coordinates of the interactive prop projection area. The interactive experience server includes a program entry module, which is used to enter the interactive program in the interactive device.

[0013] As a further description of the above technical solution:

[0014] The gesture posture recognition module trains and recognizes the gesture images acquired by the plurality of infrared camera modules based on the Al recognition module, and selects the recognition result with the highest recognition completeness and matching degree to form the gesture posture information combined with the relative position information of the interactive prop.

[0015] As a further description of the above technical solution:

[0016] The data transmission module wirelessly transmits the projection area coordinates and the gesture posture information to the interactive experience server, and the data conversion module of the interactive experience server converts the gesture posture information into index information.

[0017] As a further description of the above technical solution:

[0018] The fusion server includes a projection image entry module, which uploads and enters the interactive image data, and the index information associates the gesture posture information with the interactive image data.

[0019] As a further description of the above technical solution:

[0020] The fusion server includes a coordinate integration module, an image processing module, and an image fusion module. The coordinate integration module integrates and summarizes the projection area coordinates and interactive image data corresponding to the plurality of interactive props.

[0021] As a further description of the above technical solution:

[0022] The image processing module performs distortion correction processing corresponding to the interactive image data based on the projection area coordinates of the projection interaction module and each interactive prop, so that the image projected onto the interactive prop matches the projection area to form corrected image data. The image fusion module performs projection layer superposition and edge fusion processing on each of the corrected image data to form fused image data, and projects the fused image data to the interactive experience module through the projection interaction module.

[0023] In summary, due to the adoption of the above technical solution, the present invention has the following beneficial effects compared with the prior art:

[0024] The object recognition device based on infrared images of the present invention integrates relevant infrared detection equipment and is used for object recognition and position detection of multiple categories in the immersive space. The audience can interact with the immersive interactive game by moving multiple types of preset objects in the space, thereby enriching the game entertainment of the immersive space. When in use, the infrared camera will obtain the infrared image of the interactive props placed by the user on the interactive desktop in real time; after the relative position of the interactive desktop and props is recognized by the detection main server, the three-dimensional models of the two are placed in the three-dimensional coordinate system, and the coordinates of the projection area on the interactive props are determined by the spatial position of the two, thereby realizing the establishment of the coordinates of the projection area on the interactive props; through the control association of the interactive experience server, the fusion server queries and fuses the interactive images on each interactive prop and desktop, and projects them to the interactive experience module through the projection interaction module; and the interactive image is associated with the recognized information of multiple user gestures and postures to achieve a more diversified and humanized interactive effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0026] Figure 1 A schematic diagram of the system structure of an interactive device based on infrared image object recognition Figure 1 .

[0027] Figure 2 A schematic diagram of the structure of an interactive device based on infrared image object recognition Figure 1 .

[0028] Figure 3 A schematic diagram of the structure of an interactive device based on infrared image object recognition Figure 2 .

[0029] Figure 4 A schematic diagram of the system structure of an interactive device based on infrared image object recognition Figure 2 .

[0030] Figure 5 The present invention is a schematic diagram of the system structure of a detection main server in an interactive device based on infrared image object recognition.

[0031] Figure 6 The present invention is a schematic diagram of the system structure of an interactive experience server in an interactive device based on infrared image object recognition.

[0032] Figure 7 The figure is a schematic diagram of the system structure of a fusion server in an interactive device based on infrared image object recognition.

[0033] Figure 8 Workflow of an interactive device based on infrared image object recognition Figure 1 .

[0034] Fig. 9 Workflow of an interactive device based on infrared image object recognition Figure 2 .

[0035] Legend:

[0036] 1. Infrared camera; 2. Projector; 3. Hidden cover; 4. Interactive desktop; 5. Interactive props; 6. Infrared bandpass filter; 7. Transparent glass; 8. Infrared fill light; 9. Infrared high-pass filter. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0039] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0041] In the description of the embodiments of the present invention, it should be noted that the terms "upper", "inner", etc. indicate directions or positional relationships based on the directions or positional relationships shown in the accompanying drawings, or are directions or positional relationships in which the inventive product is usually placed when in use. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore should not be understood as a limitation on the present invention.

[0042] In the description of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a direct connection, or an indirect connection through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0043] See also Figure 1-9 The present invention provides a technical solution: an interactive device based on infrared image object recognition, comprising:

[0044] An infrared camera module, which is used to obtain infrared images of an interactive experience module in real time. The interactive experience module includes an interactive desktop and interactive props. The infrared images include an interactive desktop image, an interactive prop image, and a gesture image of a person.

[0045] The detection main server includes a prop recognition module, a coordinate simulation module, a gesture recognition module and a data transmission module. The prop recognition module recognizes the relative position information of the interactive desktop and the interactive props on the acquired infrared image. The coordinate simulation module puts the pre-entered interactive prop information model into the corresponding position of the three-dimensional coordinate system with the interactive desktop model based on the relative position information, and obtains the projection area coordinates of the interactive prop. The gesture recognition module recognizes the gesture image to form gesture information.

[0046] Interactive experience server, which is used for interactive program control and operation;

[0047] A fusion server processes the corresponding interactive image data based on the projection area coordinates and the gesture posture information, and projects the data onto the interactive experience module through a projection interaction module.

[0048] like Figure 2-4 As shown, the equipment layout corresponding to the device is specifically as follows: the detection main server, interactive experience server, and fusion server are all connected to the router, and the above four are arranged in the computer room, the infrared camera module corresponds to the infrared camera 1, the projection interaction module corresponds to the projector 2, the infrared camera 1 and the projector 2 are arranged equidistantly along the edge of the hidden cover 3, and the hidden cover 3 is provided with light holes corresponding to the two. In addition, an infrared bandpass filter 6 is provided at the light hole corresponding to the infrared camera 1, and a transparent glass sheet 7 is provided at the light hole corresponding to the projector 2. Infrared fill lights 8 are also provided around the two, and infrared high-pass filters 9 are provided at the light holes corresponding to the infrared fill lights 8.

[0049] The infrared camera 1 can collect visible light source and infrared light source signals at the same time. Here, after the visible light is filtered out by the infrared bandpass filter 6, only the infrared light source is collected, avoiding the interference of external light and projection light on the detection. In one embodiment, the device includes 6 4mm fixed interactive infrared cameras;

[0050] The infrared bandpass filter 6 allows only infrared light in the 850nm band to pass through, and cuts off visible light and other light bands, so as to ensure that the reflected light after the infrared light irradiates the object can be successfully captured by the infrared camera 1, while ensuring that the camera is not affected by lighting and projection light;

[0051] Projector 2 is used for interactive projection, projecting real-time interactive game screens, which are fused by the fusion server and output to projector 2. In one embodiment, the device includes four 6000 lumen laser projectors;

[0052] Wherein, the transparent glass sheet 7 is installed at the front end of the lens of the projector 2 to protect the lens of the projector 2;

[0053] Infrared fill light 8, used to compensate for the low brightness of infrared light in the scene, affecting the detection accuracy. In one embodiment, the device is provided with 6 30W, 120 degrees, 850nm infrared fill lights;

[0054] The infrared high-pass filter 9 is used to protect and hide the infrared fill light 8. The infrared high-pass filter and the visible light low-pass filter ensure that the infrared fill light is perfectly hidden without affecting the passage of the infrared light.

[0055] The above structure is assembled on the hidden cover plate 3, which is used to hide the top equipment and the hanger and bracket lighting device, so as to maintain the beauty of the exhibition hall and also protect the equipment;

[0056] The detection main server processes the images collected by the camera in real time, detects interactive props such as plates and wine glasses, as well as gesture interaction actions, and converts the interaction data into display coordinates before transmitting them to the interactive experience server;

[0057] Interactive experience server, running interactive game programs, can interact through props such as plates and wine glasses as well as gestures;

[0058] The fusion server is used to process the interactive game screen in real time, perform distortion correction and edge fusion, and then project it onto the real desktop.

[0059] The facility objects corresponding to the interactive experience module include an interactive desktop 4 and a plurality of interactive props 5 of different styles placed on the interactive desktop 4 .

[0060] The interactive desktop 4 is the interactive tabletop of the entire system. The projector 2 projects the image onto the desktop. At the same time, props such as interactive plates and wine glasses placed on the desktop can be detected by the infrared camera 1. Users sitting at the edge of the interactive desktop 4 can adjust their gestures and postures to make the desktop display different effects. The interactive props 5 can use interactive plates, interactive wine glasses and other objects to achieve diversified interactive functions.

[0061] The object recognition device based on infrared images of the present invention integrates relevant infrared detection equipment, is used for object recognition and position detection of multiple categories in immersive space, and can be applied to immersive spaces in museums, science and technology museums, and various theme museums. The audience can interact with immersive interactive games by moving multiple types of preset objects in the space, thereby enriching the game entertainment of the immersive space. It can not only enrich the display form of the exhibition hall, but also be used in various exhibition halls as a low-cost, environmentally friendly, and reusable system. It realizes a set of object recognition system based on infrared images by combining infrared cameras, fill lights, infrared filters, mounting brackets, projectors and other equipment, combined with corresponding software algorithms. Through this system, the types of object recognition in the immersive space can be increased, the recognition accuracy can be increased, and the fun of the game can be improved.

[0062] The device is set in the interactive table area of ​​4 meters in diameter, and detects objects with a diameter of no less than 8 cm. It can provide interaction for no less than 15 people and no less than 3 forms of object interaction at the same time. The object detection response time is less than 100ms.

[0063] The detection main server also includes a scanning and system building module, which scans the appearance of the interactive desktop and the projection interactive module based on the infrared camera module and builds the three-dimensional coordinate system with the models of the two. This design makes the subsequent position recognition of the interactive props on the interactive desktop more accurate, so as to improve the accuracy of its local projection area.

[0064] The detection main server also includes a prop entry module, through which the interactive prop information is entered for use in the coordinate simulation module to obtain the coordinates of the interactive prop projection area, and the interactive experience server includes a program entry module, which is used to enter the interactive program in the interactive device. By entering a variety of interactive prop models, the identification of the types of interactive props in subsequent images, the detection of their relative positions with the interactive desktop, and the construction of the three-dimensional models of the two in the three-dimensional coordinate system are more accurate, resulting in a better interactive effect. In addition, the interactive mode of the device can be edited by a program to achieve diversified and adjustable interactive effects.

[0065] The gesture recognition module trains and recognizes the gesture images acquired by the infrared camera module based on the Al recognition module, and selects the recognition result with the highest recognition completeness and matching degree to form gesture information combined with the relative position information of the interactive prop. Through Al training and recognition, the user's interaction intention is obtained more accurately, and the interaction effect is more humanized.

[0066] The data transmission module wirelessly transmits the projection area coordinates and the gesture posture information to the interactive experience server, and the data conversion module of the interactive experience server converts the gesture posture information into index information.

[0067] The fusion server includes a projection image input module, which uploads and inputs the interactive image data, and the index information associates the gesture information with the interactive image data, thereby making the information transmission between device modules more efficient and the retrieval query of the interactive image data more time-saving.

[0068] The fusion server includes a coordinate integration module, an image processing module, and an image fusion module. The coordinate integration module integrates and summarizes the projection area coordinates and interactive image data corresponding to the plurality of interactive props.

[0069] The image processing module performs distortion correction processing corresponding to the interactive image data based on the projection area coordinates of the projection interaction module and each interactive prop, so that the image projected onto the interactive prop matches the projection area to form corrected image data. The image fusion module performs projection layer superposition and edge fusion processing on each of the corrected image data to form fused image data, and projects the fused image data to the interactive experience module through the projection interaction module.

[0070] The working principle of an interactive device based on infrared image object recognition in this embodiment includes:

[0071] S1, the scanning and system building module uses an infrared scanner to scan the interactive desktop and the hidden cover plate, and establishes a three-dimensional coordinate system with the two. The scanning mechanism is to measure the distance through the time interval between the emission and reception of infrared rays to complete the three-dimensional scanning;

[0072] S2. Input basic information of the device. The prop input module inputs the size and appearance information of the interactive props. The uploaded data information is a 3D model with accurate size. The program input module inputs the system running program. The projection image input module uploads and inputs the interactive image data.

[0073] S3, the infrared camera module acquires the interactive desktop, interactive props and gesture images of personnel in real time; the object recognition device based on infrared images displays different contents in real time by detecting the types and positions of various interactive props in real time, and can also detect gestures and display different effects according to different gestures;

[0074] S4, the prop recognition module recognizes and corrects the relative position information of the interactive desktop and the interactive props on the acquired infrared image; the correction is mainly based on the size deviation caused by the shooting angle;

[0075] S5. The coordinate simulation module integrates the interactive prop model into the corresponding position of the three-dimensional coordinate system based on the input interactive prop information and relative position information, and obtains the projection area coordinates of the interactive prop; locates the interactive desktop and the interactive prop image after correction, and places the interactive prop in the three-dimensional coordinate system based on the interactive desktop model size and the relative position information in the image, and determines the coordinates of the projection area through the relative position of the projection area on the interactive prop, so as to ensure the accuracy of the subsequent projection range control on the physical object;

[0076] S6. The gesture and posture recognition module performs training recognition on the gesture images acquired by multiple infrared camera modules based on Al recognition, and selects the recognition result with the highest recognition completeness and matching degree to form gesture and posture information combined with the relative position information of the interactive props; due to the differences in the layout angles and orientations of different infrared camera modules, the acquired infrared images may have the problem of unclear recognition of user gestures and postures, so the Al recognition module performs multi-angle and image training recognition, selects the recognition result with the highest completeness of gesture and posture images and the highest matching degree with the corresponding standard gesture and posture, to ensure accurate recognition of the user's interaction intention;

[0077] S7, the data transmission module wirelessly transmits the projection area coordinates and the gesture posture information to the interactive experience server, and converts the gesture posture information into index information;

[0078] S8, the coordinate integration module integrates the projection area coordinates and index information corresponding to the multiple interactive props, and matches each interactive prop with the interactive image data based on the index information; thereby making it faster and more efficient to obtain the projection interactive image corresponding to the user's gestures and postures;

[0079] S9, the image processing module performs distortion correction processing on the interactive image data based on the projection interaction module and the projection area coordinates of each interactive prop, so as to make it coincide with the projection area and form corrected image data;

[0080] S10, the image fusion module performs projection layer superposition and edge fusion processing on each corrected image data to form fused image data, and projects the fused image data to the interactive experience module through the projection interaction module. In this way, one-time interactive image projection of multiple interactive props on the interactive desktop can be achieved. The device display platform is a desktop with a diameter of 4m. The desktop is white and has no pattern. All display content is displayed by the projector. The projector projects the interactive screen onto the desktop and displays different content in real time according to the position of the interactive props detected by the detection server. For example, some small animations of meals are displayed in the plate props, and the fluctuation animation of wine is displayed in the wine glass props.

[0081] In summary, due to the adoption of the above technical solution, the interactive device based on infrared image object recognition in this embodiment has the following beneficial effects compared with the prior art:

[0082] The object recognition device based on infrared images of the present invention integrates relevant infrared detection equipment and is used for object recognition and position detection of multiple categories in the immersive space. The audience can interact with the immersive interactive game by moving multiple types of preset objects in the space, thereby enriching the game entertainment of the immersive space. When in use, the infrared camera will obtain the infrared image of the interactive props placed by the user on the interactive desktop in real time; after the relative position of the interactive desktop and props is recognized by the detection main server, the three-dimensional models of the two are placed in the three-dimensional coordinate system, and the coordinates of the projection area on the interactive props are determined by the spatial position of the two, thereby realizing the establishment of the coordinates of the projection area on the interactive props; through the control association of the interactive experience server, the fusion server queries and fuses the interactive images on each interactive prop and desktop, and projects them to the interactive experience module through the projection interaction module; and the interactive image is associated with the recognized information of multiple user gestures and postures to achieve a more diversified and humanized interactive effect.

[0083] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. An interactive device based on infrared image object recognition, characterized in that: include: An infrared camera module, which is used to obtain infrared images of an interactive experience module in real time. The interactive experience module includes an interactive desktop and interactive props. The infrared images include an interactive desktop image, an interactive prop image, and a gesture image of a person. The detection main server includes a prop recognition module, a coordinate simulation module, a gesture recognition module and a data transmission module. The prop recognition module recognizes the relative position information of the interactive desktop and the interactive props on the acquired infrared image. The coordinate simulation module puts the pre-entered interactive prop information model into the corresponding position of the three-dimensional coordinate system with the interactive desktop model based on the relative position information, and obtains the projection area coordinates of the interactive prop. The gesture recognition module recognizes the gesture image to form gesture information. Interactive experience server, which is used for interactive program control and operation; A fusion server processes the corresponding interactive image data based on the projection area coordinates and the gesture posture information, and projects the data onto the interactive experience module through a projection interaction module.

2. The interactive device based on infrared image object recognition according to claim 1, characterized in that: The detection main server also includes a scanning and system building module. The scanning and system building module scans the appearance of the interactive desktop and the projection interactive module based on the infrared camera module, and builds the three-dimensional coordinate system with models of the two.

3. The interactive device based on infrared image object recognition according to claim 1, characterized in that: The detection main server also includes a prop entry module, through which the interactive prop information is entered for use in the coordinate simulation module to obtain the coordinates of the interactive prop projection area. The interactive experience server includes a program entry module, which is used to enter the interactive program in the interactive device.

4. The interactive device based on infrared image object recognition according to claim 1, characterized in that: The gesture posture recognition module trains and recognizes the gesture images acquired by the plurality of infrared camera modules based on the AI ​​recognition module, and selects the recognition result with the highest recognition completeness and matching degree to form gesture posture information combined with the relative position information of the interactive prop.

5. The interactive device based on infrared image object recognition according to claim 1, characterized in that: The data transmission module wirelessly transmits the projection area coordinates and the gesture posture information to the interactive experience server, and the data conversion module of the interactive experience server converts the gesture posture information into index information.

6. The interactive device based on infrared image object recognition according to claim 5, characterized in that: The fusion server includes a projection image entry module, which uploads and enters the interactive image data, and the index information associates the gesture posture information with the interactive image data.

7. The interactive device based on infrared image object recognition according to claim 1, characterized in that: The fusion server includes a coordinate integration module, an image processing module, and an image fusion module. The coordinate integration module integrates and summarizes the projection area coordinates and interactive image data corresponding to the plurality of interactive props.

8. The interactive device based on infrared image object recognition according to claim 7, characterized in that: The image processing module performs distortion correction processing corresponding to the interactive image data based on the projection area coordinates of the projection interaction module and each interactive prop, so that the image projected onto the interactive prop matches the projection area to form corrected image data. The image fusion module performs projection layer superposition and edge fusion processing on each of the corrected image data to form fused image data, and projects the fused image data to the interactive experience module through the projection interaction module.

Images