A scientific knowledge display system based on multi-modal data fusion technology

Abstract

The present application relates to a kind of scientific knowledge display system based on multi-modal data fusion technology, belong to digital information technology field.The display system includes full week display component, carrier, interactive component and central control unit.Full week display component is hollow pipeline structure, is formed by multiple spliced curved surface display unit, for showing digital content in panorama form.Viewers ride in adjustable posture carrier, while watching the display content in pipeline, interactive component collects the multi-modal data of audience.Central control unit runs artificial intelligence model, carries out fusion analysis to the multi-modal information collected, and based on audience age estimation, emotional bias, interactive instruction etc.dynamically adjusts the difficulty of display content, information density and display rhythm.The system can realize the real-time self-adaptation of scientific knowledge content and audience demand, especially suitable for astronomy, ocean, prehistoric environment etc.large science popularization display scene.

Description

Technical Field

[0001] This invention belongs to the field of digital information technology, and more specifically, relates to a scientific knowledge display system based on multimodal data fusion technology. Background Technology

[0002] Existing scientific knowledge display systems mostly use flat-panel displays, projection screens, or fixed interactive terminals to play content. Viewers typically passively receive pre-set content, lacking the ability to dynamically adjust the content based on group characteristics and real-time feedback. Furthermore, existing systems are largely limited to simple touchscreen input or single voice commands in terms of multimodal interaction, failing to comprehensively analyze multimodal information such as audience voice expressions, posture changes, and emotional responses. Consequently, they cannot personalize the difficulty, information density, and pacing of the content during the presentation. Especially for large-scale digital displays of astronomical, marine, and prehistoric forest content, traditional two-dimensional screens and conventional projection methods cannot provide a good immersive experience, failing to create a sense of surround sound, spatial depth, and a truly immersive atmosphere. Therefore, there is an urgent need for a new system based on multimodal data fusion and a surround display structure to achieve real-time adaptation between scientific knowledge content and audience needs, thereby improving immersion, interactivity, and the effectiveness of science education.

[0003] A review of relevant publicly available technologies reveals several key solutions. CN110824971B proposes a system for showcasing the effects of sleep products, using a simulated demonstration program to showcase the user experience of these products. JP3229803U presents a magnetic display stand, which allows for the display of different content by adjusting its unfolding degree. US20150026010A1 proposes a product display system that captures the user's level of interest while viewing products, thereby planning a corresponding browsing route.

[0004] The above technical solutions all propose various systems or devices for displaying goods or content, but there are currently few mentions of solutions for immersive science popularization displays, especially for immersive solutions that require displaying large scenes.

[0005] The foregoing description of the background art is intended only to facilitate understanding of the invention. This description does not endorse or acknowledge any common general knowledge in the materials mentioned. Summary of the Invention

[0006] The purpose of this invention is to provide a scientific knowledge display system based on multimodal data fusion technology. The system includes a panoramic display component, a vehicle, interactive components, and a central control unit. The panoramic display component is a hollow pipe structure composed of multiple spliced ​​curved display units, used to display digital content in a panoramic format. Viewers sit in an adjustable vehicle and, while viewing the content displayed within the pipe, the interactive components collect multimodal data from the viewers, including controller pointing data, voice information, posture, and head orientation information. The central control unit runs an artificial intelligence model to fuse and analyze the collected multimodal information, and dynamically adjusts the difficulty, information density, and display pace of the content based on estimated viewer age, emotional bias, and interactive commands. This system can achieve real-time adaptation between scientific knowledge content and viewer needs, and is particularly suitable for large-scale science popularization display scenarios such as astronomy, oceanography, and prehistoric environments.

[0007] This invention adopts the following technical solution: a scientific knowledge display system based on multimodal data fusion technology, the display system comprising:

[0008] A full-circumference display component, wherein the full-circumference display component is a hollow pipe structure, the full-circumference display component includes a frame and a display surface covering the inner surface of the frame; the display surface is formed by splicing together several curved display units with a certain curvature, each curved display unit including a curved display screen and a driving circuit for driving the curved display screen;

[0009] The vehicle is located inside the omnidirectional display assembly. The vehicle includes a passenger cabin, multiple seats, and a bottom support base. The support base includes multiple independently controlled support components for adjusting the attitude of the vehicle.

[0010] An interactive component is used to capture multimodal data of the audience and send the multimodal data to the central control unit;

[0011] The central control unit is used to control the content displayed on the display surface based on multimodal information input by the audience;

[0012] The audience sits in the cabin, and the display system plays content in a full-circle display format through the full-circle display component. During this process, the system collects multiple multimodal feedback information from the user to adjust the display content in real time.

[0013] Preferably, the support component is a telescopic rod array consisting of multiple telescopic rods; each telescopic rod includes a transition support and is fixedly connected to the bottom of the vehicle through a flexible connection structure to achieve attitude control of the vehicle.

[0014] Preferably, the interactive component includes a controller and a receiving component; the controller includes at least one transmitting module for transmitting light signals or ultrasonic signals to indicate the location of a display surface area currently selected by the viewer; the receiving component is disposed on or below the surface of the display surface for receiving signals transmitted by the controller and resolving the viewer's pointing position and command type.

[0015] Preferably, the interactive component further includes a receiving component B; the receiving component B includes multiple sensors for collecting audience voice information; and also includes a communication element for transmitting the collected voice information to the central control unit.

[0016] Preferably, the interactive component further includes a receiver component; the receiver component includes multiple three-dimensional depth sensors for collecting the head position, sitting height and orientation information of each audience member, and the central control unit adjusts the height of the vehicle or the viewing angle focus of the content displayed in the omnidirectional display component according to the information.

[0017] Preferably, the central control unit operates an artificial intelligence model with multimodal processing capabilities; the artificial intelligence model at least classifies the audience's voice information, including at least impression expressions, discussion, invalid or noise, and questions or verifications, and dynamically adjusts the difficulty, rhythm, or information density of the displayed content based on the analysis results.

[0018] Preferably, the central control unit further includes, based on the collected multimodal information of the audience, analyzing the average age group indicated by the average head height of the audience, the audience's emotional expression and the positivity of their comments, as well as the audience's control commands, and dynamically adjusting the display rhythm, information density and content depth of the displayed content.

[0019] The beneficial effects achieved by this invention are:

[0020] This technical solution combines a fully immersive curved display component with an adjustable vehicle structure to achieve a three-dimensional and immersive display of large-scale scientific scenes, overcoming the problem of insufficient immersion caused by traditional two-dimensional flat screen displays.

[0021] This technical solution utilizes multimodal sensing devices to comprehensively collect audience voice expressions, controller pointing data, head orientation and posture information, and combines them with artificial intelligence big data model analysis. Based on audience characteristics and real-time feedback, it can dynamically adjust the difficulty, information density and presentation rhythm of the displayed content, thereby improving audience participation and experience accuracy.

[0022] The central control unit described in this technical solution establishes a multimodal fusion analysis algorithm framework, enabling the system to adaptively schedule visual effects, interactive modules, and narrative structure during the presentation process, providing highly personalized presentation strategies and effectively enhancing the learning interest and immersive experience of different groups such as teenagers.

[0023] The hardware and software components of the demonstration system of this technical solution adopt a modular design. Each working module and component of the hardware component, as well as the instructions, parameters and algorithms of the software component, can be easily replaced and / or upgraded later, thereby reducing the construction and maintenance costs of this system. Attached Figure Description

[0024] The invention will be further understood from the following description taken in conjunction with the accompanying drawings. The components in the drawings are not necessarily drawn to scale, but rather the emphasis is on illustrating the principles of the embodiments. In different views, the same reference numerals designate corresponding parts.

[0025] Reference numerals: 100 - Full-circle display component; 110 - Frame; 112 - Curved display unit; 114 - Curved display screen; 116 - Drive circuit; 120 - Display control unit; 200 - Carrier; 210 - Support base; 212 - Telescopic rod; 214 - Adapter bracket; 300 - Interactive component; 400 - Central control unit; 700 - Computer system; 702 - Bus; 704 - Processor; 706 - Main memory; 708 - Read-only memory; 710 - Storage device; 712 - Display; 714 - Input device; 716 - Cursor control device; 718 - Network device;

[0026] Figure 1 This is a schematic diagram of the framework of the display system described in this invention;

[0027] Figure 2 This is a schematic diagram of the layout of the display system described in this invention;

[0028] Figure 3 This is a schematic diagram of the architecture of the full-circle display component described in an embodiment of the present invention;

[0029] Figure 4 This is a schematic diagram of the architecture of the curved surface display unit described in an embodiment of the present invention;

[0030] Figure 5 This is a schematic diagram illustrating the configuration of the vehicle described in an embodiment of the present invention;

[0031] Figure 6 This is a schematic diagram of the telescopic rod array described in an embodiment of the present invention;

[0032] Figure 7 This is a schematic diagram of the computer system framework used in the central control unit described in this embodiment of the invention. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to its embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the invention. Other systems, methods, and / or features of this embodiment will become apparent to those skilled in the art after reviewing the following detailed description. All such additional systems, methods, features, and advantages are intended to be included within this specification, within the scope of the invention, and protected by the appended claims. Further features of the disclosed embodiments are described in the following detailed description, and these features will become apparent from the following detailed description.

[0034] In the accompanying drawings of this invention, the same or similar reference numerals correspond to the same or similar components. In the description of this invention, it should be understood that if terms such as "upper," "lower," "left," and "right" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or component referred to must have a specific orientation. Because the invention is constructed and operated in a specific orientation, the terms describing positional relationships in the drawings are for illustrative purposes only and should not be construed as limiting this patent. Those skilled in the art can understand the specific meaning of the above terms according to the specific circumstances.

[0035] Example 1: Exemplary, as shown in the appendix Figure 1 As shown, a scientific knowledge display system based on multimodal data fusion technology is proposed, the display system comprising:

[0036] A full-circumference display component 100 is a hollow pipe structure. The full-circumference display component includes a frame and a display surface covering the inner surface of the frame. The display surface is formed by splicing together several curved display units with a certain curvature. Each curved display unit includes a curved display screen and a driving circuit for driving the curved display screen.

[0037] The vehicle 200 is disposed inside the omnidirectional display assembly. The vehicle includes a passenger cabin, multiple seats, and a bottom support base. The support base includes multiple independently controlled support components for adjusting the attitude of the vehicle.

[0038] Interactive component 300 is used to capture multimodal data of the audience and send the multimodal data to the central control unit;

[0039] The central control unit 400 is used to control the content displayed on the display surface based on multimodal information input by the audience;

[0040] The audience sits in the cabin, and the display system plays content in a full-circle display format through the full-circle display component. During this process, the system collects multiple multimodal feedback information from the user to adjust the display content in real time.

[0041] Preferably, the support component is a telescopic rod array consisting of multiple telescopic rods; each telescopic rod includes a transition support and is fixedly connected to the bottom of the vehicle through a flexible connection structure to achieve attitude control of the vehicle.

[0042] Preferably, the interactive component includes a controller and a receiving component; the controller includes at least one transmitting module for transmitting light signals or ultrasonic signals to indicate the location of a display surface area currently selected by the viewer; the receiving component is disposed on or below the surface of the display surface for receiving signals transmitted by the controller and resolving the viewer's pointing position and command type.

[0043] Preferably, the interactive component further includes a receiving component B; the receiving component B includes multiple sensors for collecting audience voice information; and also includes a communication element for transmitting the collected voice information to the central control unit.

[0044] Preferably, the interactive component further includes a receiver component; the receiver component includes multiple three-dimensional depth sensors for collecting the head position, sitting height and orientation information of each audience member, and the central control unit adjusts the height of the vehicle or the viewing angle focus of the content displayed in the omnidirectional display component according to the information.

[0045] Preferably, the central control unit operates an artificial intelligence model with multimodal processing capabilities; the artificial intelligence model at least classifies the audience's voice information, including at least impression expressions, discussion, invalid or noise, and questions or verifications, and dynamically adjusts the difficulty, rhythm, or information density of the displayed content based on the analysis results.

[0046] Preferably, the central control unit further includes, based on the collected multimodal information of the audience, analyzing the average age group indicated by the average head height of the audience, the audience's emotional expression and the positivity of their comments, as well as the audience's control commands, and dynamically adjusting the display rhythm, information density and content depth of the displayed content.

[0047] As attached Figure 2 The illustration shows an exemplary implementation of the display system.

[0048] For example, in a preferred embodiment, the circumferential display component is a hollow pipe that completely covers the vehicle. The interior of the pipe serves as an operational medium for content display and interaction with the audience. The circumferential display component includes a frame and a display surface covering the inner surface of the frame.

[0049] Preferably, the frame is a hollow cylindrical structure, extending along the central axis with the initial position of the carrier as the central axis to form a display channel. The cross-section of the hollow cylindrical structure formed by the frame perpendicular to the central axis can be a perfect circle or an ellipse with a certain curvature, and the part of the frame that contacts the ground is an open structure, with the two edges of the open structure forming an angle of approximately 300° or greater with the central axis, so that except for the bottom area of ​​the carrier, the rest is a continuous display area.

[0050] The inner arc wall of the frame forms the display surface. The frame is externally supported by a lightweight rigid framework to create a complete curved surface support, ensuring sufficient rigidity, continuity, and stability of the overall curved surface after multiple curved display units 112 are installed. Preferably, the framework material can be an aluminum-magnesium alloy truss, a carbon fiber frame, or a composite panel mesh structure.

[0051] Preferred options are listed below. Figure 3 and attached Figure 4 The diagram illustrates an exemplary architecture of the full-circle display component 100. The display surface is formed by splicing together several curved display units 112 with a certain curvature. Each curved display unit 112 includes a curved display screen 114. The curved display screen 114 can be an LED display screen, a laser rear projection screen, an OLED flexible screen, or other screen types. The curvature of the curved display screen 114 can be specifically designed according to the curvature of the frame. The display resolution of the curved display screen 114 is preferably P2.5 or a higher pixel density level. Each module is combined with a seamless splicing slot to form a continuous curved imaging surface. The modules are connected by magnetic latches or track sliding methods for easy disassembly and maintenance.

[0052] Furthermore, each of the display units includes a driving circuit 116 for driving the curved display screen. The driving circuit includes a signal receiving module, an image processing module, a timing control module, and a power control module, used to locally decode, refresh, and adjust the brightness of image data from the central control unit 400, thereby realizing unit-level image presentation control.

[0053] The signal receiving module receives image subframe data for the corresponding display area from the image signal distributor; the image processing module performs format conversion, color restoration, and bitmap reconstruction on the received data to adapt to the pixel arrangement and spatial curvature requirements of curved screens; the timing control module controls the scanning timing and refresh rate of this display unit to keep it consistent with the global screen refresh rate of the system; and the power control module dynamically adjusts the driving voltage and current of LED / OLED and other light-emitting elements according to the brightness distribution of the content to improve display efficiency and suppress heat generation.

[0054] Preferably, the driving circuit 116 further works in conjunction with the display control unit 120 to control the overall display surface to present a unified display image fusion effect.

[0055] For example, the display control unit 120 includes acquiring content to be displayed from the central control unit 400, which may include information such as video, still images, music, and text, and may be a combination of the above information.

[0056] For example, the image signal distributor of the display control unit 120 divides the image source into regions according to the spatial mapping relationship of the panoramic content, and sends the image frame information of different regions to the driving circuit of the corresponding display unit respectively; preferably, the receiving image frame information is decoded, reconstructed in format and interpolated by each driving circuit to ensure seamless integration and color consistency of the picture under the curved splicing structure.

[0057] To further optimize the splicing display effect, each driving circuit 116 also includes an algorithm for executing curved image correction. Based on a preset screen space perspective transformation model or a curvature compensation matrix generated through sampling calibration, the algorithm corrects the image in the edge region, eliminating geometric distortion, edge stretching, or ghosting. In some preferred embodiments, the driving circuits also include a synchronization control circuit to maintain inter-frame timing consistency, ensuring that screen tearing, stuttering, or synchronization delays do not occur during high dynamic content playback.

[0058] On the other hand, viewers of the displayed content can adjust the content and display process of the display system by riding in the vehicle 200 and watching the content presented by the omnidirectional display component 100. This includes changes in posture within the vehicle 200 and receiving multimodal information of the viewers during the viewing process from the multimodal sensing device on the vehicle 200.

[0059] For example, in a preferred embodiment, the carrier 200 is primarily a metal structure to provide sufficient load-bearing strength and rigidity to maintain its shape. Preferably, the exterior of the carrier 200 is covered with sheet metal, such as aluminum sheet or polymer sheet, to form a metal frame + composite shell structure. The carrier is streamlined in shape, with a selectable length of approximately 5 to 10 meters and a width of approximately 1.5 to 2 meters.

[0060] The interior of the vehicle 200 is a seating cabin. The seating cabin is equipped with multiple integrated seats so that the audience can sit in the cabin in a stable sitting posture.

[0061] Preferably, outer protective panels are provided on both sides of the seating cabin. The height of the outer protective panels is preferably at the isosceles height of the spectator in a seated position, so that the spectator's field of vision to the sides is below waist level. A protective floor is provided at the bottom of the seating cabin to isolate the electronic and mechanical components below the seating cabin from the seating cabin itself. Preferably, the top of the vehicle 200 is an open structure to fully expose the seating cabin to the outside world and provide the spectators in the seating cabin with a large field of vision.

[0062] For example, the vehicle 200 is provided with a support base 210 at its bottom. The support base 210 has multiple support components for supporting the vehicle 200 on the ground and can allow the vehicle 200 to be in various postures.

[0063] Preferably, in some embodiments, as shown in the appendix Figure 5 and attached Figure 6 As shown, the support component includes multiple independently controllable telescopic rods 212, forming a telescopic rod array. Preferably, the multiple telescopic rods 212 are in the form of hydraulic cylinders, electric lead screws, or electric push rods. The telescopic rods 212 are evenly distributed at the bottom of the vehicle and arranged in the x and y directions as shown in the figure, forming an m×n matrix arrangement. Preferably, the bottom of the telescopic rods 212 is fixedly installed on the foundation support platform, which can be a concrete foundation, a steel structure platform, or an integrated installation bracket system. Preferably, the top of the telescopic rods 212 is provided with a transition support 214, which maintains a certain degree of flexibility through a flexible connection structure. The flexible connection structure can be a combination of universal bearings, shock-absorbing washers, or sliding guide sleeves, so that the top of the telescopic rod is firmly connected to the vehicle, while preventing local stress concentration caused by rigid connections. Preferably, all telescopic rods 212 are connected to the central control unit 400 via cables or hydraulic pipes, and the control signals of each telescopic rod 212 are independent of each other and can be controlled individually or in combination; preferably, each of the telescopic rods 212 is equipped with a built-in displacement encoder or a pull-wire displacement sensor for real-time feedback of the current position and height to achieve closed-loop control.

[0064] Furthermore, in some embodiments, various attitude adjustments of the vehicle 200 can be achieved by precisely controlling the cooperation of multiple telescopic rods 212. Exemplary control methods include, but are not limited to, the following typical action modes:

[0065] Forward / backward tilt: controls the height of the longitudinal multi-row telescopic rods 212 to change linearly and gradually, for example, from the front row to the rear row, or vice versa;

[0066] Left and right tilt: Control the height difference between the horizontal multiple rows to make the left telescopic bar 212 and the right telescopic bar 212 have a certain height difference, simulating the lateral tilt of the vehicle;

[0067] Overall Lifting / Vibration: Controls all telescopic rods 212 to lift and lower synchronously, creating an overall lifting or lowering effect; combined with low-amplitude, high-speed lifting movements, vibration feedback is achieved.

[0068] Composite posture combination: Achieve complex motion trajectories such as "slope gliding" and "wave" through multi-axis combination control.

[0069] For example, multiple support components are communicatively connected to the vehicle control system; preferably, the vehicle control system may include the following modules:

[0070] The motion control module includes a storage of multiple preset action / posture modes, sequences, numbers, etc., which can be called in real time according to the current display content script.

[0071] The real-time feedback module is used to read the displacement and / or pressure sensor signals of each support column to form a feedback closed loop;

[0072] The linkage trigger interface, in conjunction with the panoramic display module and the sound effect module, can drive the vehicle to generate corresponding dynamic feedback, such as vibration, slowing down, and shaking, when specific visual / sound effect scenes occur.

[0073] The telescopic rod used as a support component in the above description is only an exemplary implementation. In other implementations, similar posture adjustment effects may be achieved by using mechanisms or devices such as multi-axis hydraulic linkages, robotic arms, and crank rocker arms. Therefore, it is not considered a limiting implementation of this technical solution.

[0074] By adjusting the supporting components and coordinating with the specific display content, including rotating the vehicle around the central axis of the frame 110 at a certain angle, the viewer's viewing angle is tilted towards a specific area of ​​the display surface. The viewer still has a fairly large field of view, but the display system can use this method to shift the viewer's attention.

[0075] Furthermore, the vehicle also includes an interactive component 300; the interactive component 300 is based on multimodal information processing technology and is used to enable personalized interactive control between the audience and the display system during the audience's viewing of the display content while riding in the vehicle, so as to enhance the immersive display experience of the display system.

[0076] In a preferred embodiment, the interactive component 300 includes a controller provided to each viewer. The controller can be held or worn by the viewer and can be controlled by the viewer to direct the direction of the controller's transmitting port. The controller has at least one transmitting module; the transmitting module is preferably a laser transmitting unit or an ultrasonic transmitting unit, used to emit invisible laser signals or focused ultrasonic signals in a target direction, so as to project the signal from the controller's transmitting end in the form of a ray to a designated location, thereby forming a pointing cursor. The pointing cursor is considered a selection area selected by the viewer on the display surface via the controller. Since laser or ultrasonic signals are invisible, they can be supplemented with visible light, which is emitted from the transmitting end simultaneously, so that the viewer can identify the specific actual position of the pointing cursor on the display surface. Preferably, the controller also includes a battery power supply unit, a signal encoding module, and a signal synchronization module, which can attach a unique identification code to the emitted laser / ultrasonic signal to identify the specific viewer from whom the signal originates, facilitating system identification and response.

[0077] Furthermore, to capture and interpret the precise position of the pointing cursor, in an exemplary embodiment, a receiving component A is applied to receive and process the controller signal. The receiving component A includes: a laser receiver array or an ultrasonic receiver array, a signal decoding module, and a data synchronization module.

[0078] For example, the laser receiver or ultrasonic microphone array includes multiple receivers. These receivers can be disposed on the surface edge of each curved display unit 112, or between the mounting gaps of two curved display units 112, exposing the receivers to space. In other embodiments, the receivers can be disposed below the surface of the display surface, adjusting the frequency of the laser or ultrasonic waves so that they can be captured by the receivers even when they slightly penetrate the surface of the display surface. Preferably, the receivers are designed in a fitted or embedded layout according to the curvature of the display screen, and are distributed equidistantly or in a matrix pattern to ensure a signal reception area covering at least a 270-degree viewing angle across the entire tubular display surface.

[0079] For example, the signal decoding module and the data synchronization module can be integrated with the driving circuit 116 in a unified module package structure, and the signal data can be transmitted to the system main control unit in real time through flexible cabling or wireless communication. The data synchronization module is used to timestamp / position synchronize with the system timing, so that the system can judge the operational significance of the audience's indicators based on the real-time picture.

[0080] For example, the controller may include one or more control buttons, touch areas, or joysticks to work with laser or ultrasonic indicators to issue different types of commands such as "select", "confirm", and "back", thereby improving the controller's operating efficiency and making the display cursor primarily used to indicate the viewer's selection area, without needing to issue operation commands.

[0081] For example, parsing the pointer generated by the controller may include the following steps: a receiver receives and decodes signals emitted by one or more controllers to obtain signal strength and audience identification; the system calculates the signal incident angle or landing point position based on the response strength or time difference of multiple receiving units; combining the geometric calibration information of the currently displayed content on the screen with a content partitioning mapping table, the coordinates of the display area actually pointed to by the audience are calculated; this pointing coordinate is bound to the additional command currently issued by the controller and transmitted to the central control unit. Finally, the central control unit realizes spatial recognition of the audience's command position and content response.

[0082] Furthermore, in a preferred embodiment, the display system further includes a second receiving component. The second receiving component includes a directional microphone positioned directly in front of each seat on the vehicle for collecting voice information emitted by the audience member seated there.

[0083] Optionally, the directional microphone can be mounted on the front edge of the seat and pointed towards the audience's face area via a bracket. The captured sound information is connected to the central control unit 400 through a data synchronization module, and the raw audio signal is transmitted to the central control unit 400 in real time.

[0084] Preferably, the central control unit 400 can be a general-purpose or patented computer system that runs a large artificial intelligence model with multimodal processing capabilities to perform speech recognition and semantic analysis, and can classify the audience's speech into at least the following categories based on the characteristics of the speech content:

[0085] Expressions of feeling: such as "So beautiful", "So shocking", "So wonderful", etc., which express the audience's emotional and intuitive feelings about the content being presented.

[0086] Audience discussion: such as "I didn't see it clearly just now, I missed it" or "Can we watch it again?", usually includes expressions of need for the pace, repetition or details of the presentation.

[0087] Invalid or Noise Category: This includes components that are irrelevant to the displayed content, cannot be identified, or are subject to environmental noise interference.

[0088] Questions or verification questions, such as "What is the principle behind this?" or "Why is it blue?", indicate that the audience wants further explanation or in-depth information.

[0089] Based on semantic classification results, the system identifies tone and emotional color in speech based on acoustic features such as pitch, speech rate, and intensity changes. This enables further analysis of emotions and needs, and generates audience status labels or preference vectors.

[0090] Furthermore, the displayed content is further adjusted based on multiple current viewer status tags or viewer preference vectors.

[0091] Furthermore, in a preferred embodiment, the display system further includes a receiver component C for detecting the posture and / or head orientation of the audience. The receiver component C collects information on the audience's head position, sitting height, and orientation, allowing the system to automatically adjust the initial height of the vehicle based on the overall characteristics of the audience group and optimize the viewing angle or focus of the displayed content accordingly, thereby enhancing the immersive experience and comfort of the young audience.

[0092] For example, the receiving component C includes multiple sets of three-dimensional depth sensors mounted on the vehicle. The three-dimensional depth sensors are preferably structured light, TOF, or stereo vision depth cameras. Each three-dimensional depth sensor includes a depth acquisition module, an infrared transmitter, an infrared receiver, and a data transmission interface, used to periodically acquire three-dimensional point cloud data or depth image data covering the audience area.

[0093] Preferably, multiple three-dimensional depth sensors are mounted on a suitable position on the vehicle via a bracket and fixing structure, such as the front, side edge, or leading edge of the vehicle, and slightly above the head height of the audience in a seated position, so as to form an effective detection field of view covering all seating areas.

[0094] Preferably, the receiving component further includes a data acquisition unit, which is electrically connected to multiple three-dimensional depth sensors and is used to receive and store the raw point cloud data or three-dimensional coordinate stream data acquired by the three-dimensional depth sensors.

[0095] Preferably, the receiving component C further includes a posture analysis and processing unit, which includes a three-dimensional coordinate extraction, head recognition and tracking algorithm, and orientation calculation module to process the raw point cloud data and extract the spatial position coordinates (e.g., three-dimensional X, Y, Z coordinates) and head orientation vector of each audience member's head.

[0096] Furthermore, the posture analysis and processing unit generates posture data records for each audience member based on the head position and orientation information, and forms an overall posture feature vector of the audience group within the system, which is used to control the posture of the vehicle 200 and the specific display content.

[0097] When the receiving component C detects a deviation between the average head height of the audience and the preset standard height, the vehicle height adjustment module drives the lifting mechanism of the support structure to adjust the overall height of the vehicle, so that the average head height of the audience is aligned with the central area of ​​the tubular display component, ensuring that the audience can obtain the best viewing experience in a natural sitting posture.

[0098] When the receiving component C detects that the overall head orientation of the audience group is biased to one side, the display content perspective adaptive module automatically adjusts the main display area of ​​the content displayed in the pipeline-type display component or enhances the display effects of the biased side area according to the audience's gaze direction, thereby enhancing immersion and interactivity.

[0099] Example 2: This example should be understood as including at least all the features of any of the foregoing examples, and further improving upon them.

[0100] In a preferred exemplary embodiment, the display system uses the interactive components, controllers, etc. to collect and analyze the multimodal information of the audience. Based on the results of multimodal information fusion, the display rhythm, content details, and interactive modules can be dynamically adjusted to further enhance the immersive experience of the young audience.

[0101] For example, the central control unit collects multiple multimodal parameters of the audience at the same or different preset periods. In some embodiments, this includes collecting:

[0102] The controller of the i-th viewer points to the data with the selected interaction command;

[0103] The spatial coordinates of the i-th viewer's head are at height h. i The output is received by component C;

[0104] The emotional bias parameter e of the i-th viewer i The output is generated by the receiving component B in conjunction with the multimodal large model speech recognition and analysis module.

[0105] The display system generates an overall audience feature vector based on the following formula:

[0106] (1) Estimating the average height of the group:

[0107] ;

[0108] Where N represents the number of currently available seats.

[0109] (2) Calculation of average sentiment bias of the group: For example, the multimodal large model outputs the value e based on the voice sentiment analysis of the i-th audience member. i e i The numerical range is [-1, +1], where -1 represents significantly negative emotion, 0 represents neutral emotion, and +1 represents positive emotion. The average group emotion bias e avg for:

[0110] .

[0111] (3) Content depth adjustment: based on the average height h of the group avg Compared with the preset standard height h std Calculate the difficulty adjustment factor ad:

[0112] ;

[0113] Where k is an adjustment constant, which can be set by statistically analyzing the height and age ratio of the general audience; when h avg <h std When ad < 1, it indicates that the current audience is relatively young, which can reduce the difficulty of the content being presented, for example, by reducing information density and the complexity of technical terms.

[0114] (4) Calculate the presentation pace delay coefficient: based on the group's average emotional bias e avg and the number of positive comments (n) r Calculate the display rhythm delay coefficient β:

[0115] ;

[0116] In the above formula, k e and k r These are the emotion weight parameter and the comment weight coefficient, respectively, which can be set appropriately by technical personnel based on the displayed content. Among them, the number of positive comments, n... r The language model can be used to analyze audience sentiment and comments while viewing the displayed content to count the number of positive comments, and update n in each monitoring period. r The value.

[0117] Furthermore, set the display rhythm T of the displayed content.

[0118] ;

[0119] T0 represents the preset display rhythm. The display rhythm refers to the overall playback speed, pause duration, transition duration, density and continuity of information appearance, and number of interactions when the display system plays, switches, and advances various digital content (including images, videos, animations, explanatory text, and interactive elements).

[0120] Preferably, the presentation rhythm may include the following elements:

[0121] Duration of a single segment or chapter: determines the length of time each display unit or scene is viewed.

[0122] The speed at which content appears and fades out: such as the time required for subtitles, knowledge point prompts, and special effects animations to appear and disappear.

[0123] Transition pause duration between chapters or scenes: The duration of buffering, prompts, or blank paragraphs set by the system when switching between different display paragraphs.

[0124] Overall information flow frequency: reflects how many core knowledge points or visual elements are presented to the audience per unit of time.

[0125] Therefore, in this presentation system, by adjusting the presentation rhythm, the same content can be presented in a slower, more paused manner (giving the audience more time to understand and discuss), or in a more compact, efficient, and faster-paced manner (adapting to the audience's high excitement and preference for a fast pace), thereby better matching the audience's emotional state, attention level, and preference characteristics.

[0126] Furthermore, calculate the information density D:

[0127] ;

[0128] D0 represents the default information density of the original content, including the number of knowledge points and the complexity of terminology. The information density can be set sufficiently rich when creating the content and then selectively played during the actual presentation to suit the audience's immediate reactions.

[0129] Through the above calculation process, the central control unit 400 periodically updates the content difficulty adjustment coefficient ad and the display rhythm delay coefficient β, and inputs the updated display parameters into the full-circle display component and display control unit 120, that is, calculates the information density D and the display rhythm T, so as to dynamically adjust the display content difficulty, terminology density and display rhythm, to ensure that the content matches the current characteristics of the audience and improve the viewing comfort and learning interest of young viewers.

[0130] In addition, the system can set a threshold: when e avg <-0.3 and (n) r When / N)>0.4, insert a "replay prompt" segment to automatically repeat key content segments from the previous segment, or supplement the presentation with a simpler explanation.

[0131] Example 3: This example should be understood as including at least all the features of any of the foregoing examples, and further improving upon them.

[0132] As attached Figure 7 The following diagram illustrates an implementation of the computer system 700 used in the central control unit 400; the computer system 700 can be applied to the data storage, calculation, and result output processes of each working module in the identification and judgment system.

[0133] For example, computer system 700 includes bus 702 or other communication mechanism for transmitting information, and one or more processors 704 coupled to bus 702 for processing information; processor 704 may be, for example, one or more general-purpose microprocessors.

[0134] The computer system 700 also includes a main memory 706, such as random access memory (RAM), cache and / or other dynamic storage devices, coupled to a bus 702 for storing information and instructions to be executed by the processor 704; the main memory 706 can also be used to store temporary variables or other intermediate information during the execution of instructions executed by the processor 704; when these instructions are stored in a storage medium accessible to the processor 704, the computer system 700 presents itself as a dedicated machine customized to perform the operations specified in the instructions;

[0135] The computer system 700 may also include a read-only memory (ROM) 708 or other static storage device coupled to the bus 702 for storing static information and instructions of the processor 704; wherein a storage device 710, such as a disk, optical disk or USB drive (flash drive), is coupled to the bus 702 for storing information and instructions.

[0136] Furthermore, the bus 702 may also include a display 712 for displaying various information, data, media, etc., and an input device 714 for allowing users of the computer system 700 to control, manipulate, and / or interact with the computer system 700.

[0137] A preferred method of interacting with the management system may be through a cursor control device 716, such as a computer mouse or a similar control / navigation mechanism;

[0138] Furthermore, the computer system 700 may also include a network device 718 coupled to the bus 702; wherein the network device 718 may include components such as wired network cards, wireless network cards, switching chips, routers, switches, etc.

[0139] Generally speaking, the terms “engine,” “component,” “system,” and “database” used in this article can refer to the logic embodied in hardware or firmware, or to a set of software instructions that may have entries and exit points, written in programming languages ​​such as Java, C, or C++; software components can be compiled and linked into executable programs and installed in dynamic link libraries, or can be written in interpreted programming languages ​​(such as BASIC, Perl, or Python); it should be understood that software components can be called from other components or from themselves, and / or can be called in response to detected events or interrupts;

[0140] Software components configured to execute on a computing device may be provided on computer-readable media, such as optical discs, digital video discs, flash drives, magnetic disks, or any other tangible media, or as digital downloads (and may be initially stored) in compressed or installable formats that require installation, decompression, or decryption prior to execution; such software code may be stored, in part or in whole, on a memory device executing the computing device; software instructions may be embedded in firmware, such as EPROM; it should also be understood that hardware components may consist of connected logic units (e.g., gates and flip-flops), and / or may consist of programmable units (e.g., programmable gate arrays or processors);

[0141] The computer system 700 includes technologies described herein that can be implemented using custom hardwired logic, one or more ASICs or FPGAs, firmware and / or program logic, which, when combined with the computer system, enables the computer system 700 to become a dedicated computing device.

[0142] According to one or more embodiments, the techniques described herein are executed by a computer system 700 in response to a processor 704 executing one or more sequences of one or more instructions contained in main memory 706; such instructions may be read into main memory 706 from another storage medium such as storage device 710; execution of the sequence of instructions contained in main memory 706 causes processor 704 to perform the processing steps described herein; in alternative embodiments, hardwired circuitry may be used in place of or in combination with software instructions.

[0143] As used herein, the term "non-transitory medium" and similar terms refer to any medium that stores data and / or instructions that enable a machine to operate in a particular manner; such non-transitory medium may include non-volatile medium and / or volatile medium; non-volatile medium includes, for example, optical discs or magnetic disks, such as storage device 710; volatile medium includes dynamic memory, such as main memory 706.

[0144] Common forms of non-transitory media include, for example, floppy disks, hard disks, solid-state drives, magnetic tapes or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with a hole pattern, RAM, PROMs and EPROMs, FLASH-EPROMs, NVRAMs, any other memory chips or cartridges and their network versions;

[0145] Non-transient media are different from transmission media, but can be used in conjunction with transmission media; transmission media participate in information transmission between non-transient media; for example, transmission media include coaxial cables, copper wires and optical fibers, including the wires that constitute bus 702; transmission media can also take the form of sound waves or light waves, such as radio waves and infrared data communication.

[0146] While this application has been described above with reference to various embodiments, it should be understood that many changes and modifications can be made without departing from the scope of this application. That is, the methods, systems, and devices discussed above are examples. Various configurations can be appropriately omitted, substituted, or added to various processes or components. For example, in alternative configurations, methods can be performed in a different order than those described, and / or various components can be added, omitted, and / or combined. Moreover, features described with respect to certain configurations can be combined in various other configurations, such as different aspects and elements of the configuration can be combined in a similar manner. Furthermore, the elements therein can be updated as the technology develops; that is, many elements are examples and do not limit the scope of this disclosure or the claims.

[0147] Specific details are provided in the specification to offer a thorough understanding of exemplary configurations, including implementations. However, configurations can be practiced without these specific details; for example, well-known circuits, processes, algorithms, structures, and techniques have been shown without unnecessary detail to avoid obscuring the configuration. This description provides only exemplary configurations and does not limit the scope, applicability, or configuration of the claims. Rather, the foregoing description of the configurations will provide those skilled in the art with an enabling description for implementing the described techniques. Various changes can be made to the function and arrangement of the elements without departing from the spirit or scope of this disclosure.

[0148] In summary, the above detailed description is intended to be illustrative rather than restrictive, and it should be understood that these embodiments are for illustrative purposes only and not for limiting the scope of protection of the invention. After reading the description of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent changes and modifications also fall within the scope defined by the claims of this invention.

Claims

1. A scientific knowledge presentation system based on multi-modal data fusion technology, characterized in that, The display system comprises: a full-circle display assembly, which is a hollow pipe structure, comprising a frame and a display surface covering the inner surface of the frame; the display surface is formed by splicing a plurality of curved display units with a certain curvature, each curved display unit comprising a curved display screen and a driving circuit for driving the curved display screen; a carrier arranged inside the full-circle display assembly, comprising a cabin, a plurality of seats and a bottom support base, the support base comprising a plurality of independently controlled support components for adjusting the attitude of the carrier; an interactive assembly for capturing multi-modal data of the audience and sending the multi-modal data to a central control unit; a central control unit for controlling the content displayed on the display surface based on the multi-modal information input by the audience; wherein the audience is seated in the cabin, and the display system plays the display content in a full-circle display form in the full-circle display assembly, during which the multi-modal feedback information of the user is collected to adjust the display content in real time; the support components are an array of telescopic rods; each telescopic rod comprises an adapter fixedly connected to the bottom of the carrier through a soft connection structure to realize attitude control of the carrier; the interactive assembly comprises a controller and a beta receiving assembly; the controller comprises at least one transmitting module for transmitting optical signals or ultrasonic signals to express the current selected area position of the audience on the display surface; the beta receiving assembly is arranged below the display surface or the surface layer of the display surface, and is used for receiving the signals transmitted by the controller and analyzing the pointing position and instruction type of the audience; the interactive assembly further comprises a gamma receiving assembly; the gamma receiving assembly comprises a plurality of three-dimensional depth sensors for collecting the head position, sitting height and orientation information of each audience member, and the central control unit adjusts the height of the carrier or the visual angle focal point of the display content in the full-circle display assembly according to the information; the central control unit runs an artificial intelligence model with multi-modal processing capability; through the artificial intelligence model, at least the voice information of the audience is classified, including at least audience experience expression, discussion, invalid or noise, and question or verification, and the difficulty, rhythm or information density of the display content is dynamically adjusted based on the analysis result; the central control unit further comprises a dynamic adjustment module for dynamically adjusting the display rhythm, information density and content depth of the display content based on the collected multi-modal information of the audience, the average age indicated by the average head height of the audience, the emotional performance and positive nature of the comments of the audience, and the control instructions of the audience; the central control unit collects: the controller pointing data of the ith audience member to the selected interactive instruction; the display system generates an overall audience feature vector based on the following calculation formula: estimated group average height: ​ ​ ​ ​ The i-th viewer head space coordinate height h i output by the prop receiving assembly Emotion biasing parameter e of the i-th audience i output by the speech recognition analysis module of the multi-modal large model in combination with the i-th receiving component ​ ​ ； Wherein N is the current effective seat number; Group average emotional bias degree calculation: according to the i th audience's speech The audio sentiment analysis output value e i The value range of e i is [-1, +1], wherein -1 represents a significantly negative emotion, 0 represents neutrality, and +1 represents a positive emotion; and the group average emotion bias degree e avg is: ； Content depth adjustment: based on average height h of the group avg with a preset standard height h std , calculate content difficulty Degree adjustment coefficient ad: ； Wherein, k is a constant, which can be set by the proportion of the height and age of the general audience; when h avg <h std ad<1, it indicates that the current audience group is low in age, so the difficulty of the display content can be reduced; Calculate the presentation rhythm delay coefficient: based on the group average emotional bias degree e avg And the number of positive review type n r , a presentation rhythm delay coefficient β is calculated: ； In the above formula, k e and k r are respectively an emotional weight parameter and a comment weight coefficient, which can be set by a technician according to the display content; wherein the number of positive comment classes n r can be analyzed by a language large model according to the emotions of the audience and the comments when watching the display content, so as to count the number of positive comments, and update the value of n r in each monitoring period. Further, set the display rhythm T of the display content; ； Wherein T0 is the preset display rhythm; Calculate the information density D: ； Wherein D0 is the default information density under the original display content, including the number of knowledge points, the complexity of terms; The information density can be set to be rich enough when the display content is produced, and selective playing is carried out in actual display to respond to the immediate performance of the audience.

Images