Intelligent telescope control method and system based on multi-modal fusion perception

Abstract

This invention discloses a control method and system for an intelligent telescope based on multimodal fusion perception, belonging to the field of intelligent optical observation technology. The method involves: assessing the scene complexity of the target observation scene based on real-time environmental parameters, and controlling the intelligent telescope to perform target recognition at the device end or edge end according to the scene complexity; extracting visual features from visual image data and acoustic features from environmental audio data, and adjusting the fusion weights of visual and acoustic features according to real-time environmental parameters to obtain visual feature target weights and acoustic feature target weights; then weighted fusion of visual and acoustic features to obtain multimodal fusion features; inputting the multimodal fusion features into a preset multimodal fusion model, and outputting the target recognition result. Therefore, by implementing this invention, it is possible to adaptively adjust the multimodal data fusion strategy according to environmental changes, thereby improving the target recognition accuracy of the intelligent telescope.

Description

Technical Field

[0001] This invention relates to the field of intelligent optical observation technology, and in particular to an intelligent telescope control method and system based on multimodal fusion sensing. Background Technology

[0002] Telescopes are evolving from single optical observation tools to integrated and intelligent ones. Mainstream intelligent telescopes generally integrate functions such as computer vision recognition, augmented reality display, and GPS. However, in outdoor observation environments with variable lighting, complex backgrounds, and strong noise interference, traditional intelligent telescope systems struggle to maintain stable and accurate target recognition performance in complex and dynamic scenarios, especially when the observed targets are similar in appearance or when environmental conditions are harsh. Therefore, improving the target recognition accuracy of intelligent telescopes has become a pressing technical challenge.

[0003] Currently, existing technologies primarily employ a combination of pre-built image recognition models and cloud interfaces for target identification via a single visual image input, supplemented by AR head-up display modules and GPS positioning sensors. However, this approach heavily relies on pre-built image recognition models, lacking adaptability to complex environments and exhibiting low accuracy in scenarios involving similar-looking species or complex terrain. Furthermore, existing technologies utilize fixed-threshold recognition algorithms, which suffer from significant performance degradation under backlight, low-light conditions, or highly interfering backgrounds. It is evident that existing technologies offer limited interaction methods, failing to respond to user voice commands or provide personalized services based on historical observation behavior. Moreover, their limited local computing power, coupled with reliance on cloud computing, makes them susceptible to network fluctuations and lacks adaptability to complex environments, hindering their ability to meet the demands of real-time outdoor observation and resulting in poor target identification accuracy for smart telescopes. Summary of the Invention

[0004] This invention proposes a control method and system for intelligent telescopes based on multimodal fusion sensing. It addresses the problem of poor target recognition accuracy in existing intelligent telescopes due to their reliance on pre-set image recognition models and lack of adaptability to complex environments. This invention achieves adaptive adjustment of the multimodal data fusion strategy according to environmental changes, thereby improving the target recognition accuracy of the intelligent telescope.

[0005] To achieve the above objectives, this invention provides a smart telescope control method based on multimodal fusion perception, applied to a smart telescope comprising a device end and an edge end. The smart telescope control method based on multimodal fusion perception includes: acquiring visual image data, environmental audio data, and real-time environmental parameters of a target observation scene; evaluating the scene complexity of the target observation scene based on the real-time environmental parameters, and controlling the smart telescope to perform a target recognition process at the device end or edge end according to the scene complexity; performing the target recognition process: extracting visual features from the visual image data, extracting voiceprint features from the environmental audio data, and adjusting the fusion weights of the visual features and the voiceprint features according to the real-time environmental parameters to obtain visual feature target weights and voiceprint feature target weights; performing weighted fusion of the visual features and the voiceprint features based on the visual feature target weights and the voiceprint feature target weights to obtain multimodal fusion features; inputting the multimodal fusion features into a preset multimodal fusion model, outputting the target recognition result, and ending the target recognition process.

[0006] This invention proposes a control method for an intelligent telescope based on multimodal fusion perception. It collects visual image data, environmental audio data, and real-time environmental parameters to obtain multi-source data features. Then, it combines the real-time environmental parameters to assess the scene complexity of the target observation scenario, generating different control strategies to control the intelligent telescope to perform the target recognition process on devices or edge devices with varying computing power. During target recognition, the fusion weights of visual and acoustic features are adaptively adjusted based on real-time environmental parameters to obtain weighted fusion features adapted to different environments. Finally, the multimodal fusion features are input into a preset multimodal fusion model to output reliable target recognition results. Thus, by eliminating the information blind spots of a single modality through the complementary input of multi-source data features, dynamically allocating computing power according to the scene to ensure real-time performance and reliability of recognition, and adaptively adjusting the fusion weights of visual and acoustic features according to environmental parameters to avoid recognition deviations caused by fixed weights in extreme environments, the method achieves adaptive adjustment of the multimodal data fusion strategy according to environmental changes, thereby improving the target recognition accuracy of the intelligent telescope.

[0007] Furthermore, the acquisition of visual image data, environmental audio data, and real-time environmental parameters of the target observation scene includes: continuously acquiring real-time video streams of the target observation scene; performing image signal processing on the visual image data to obtain visual image data; acquiring omnidirectional audio signals within a preset observation range; performing environmental wind noise and electromagnetic interference suppression processing on the environmental audio data to obtain environmental audio data; and acquiring light intensity, environmental noise intensity, and temperature and humidity data of the target observation scene based on a preset period, and performing outlier filtering and normalization processing on the light intensity, environmental noise intensity, and temperature and humidity data respectively to obtain real-time environmental parameters.

[0008] In the above scheme, image signal processing is performed on visual images to optimize imaging quality and visual recognition, environmental wind noise and electromagnetic interference are suppressed on audio signals to increase the effective information ratio of audio signals, and outlier filtering and normalization are performed on environmental parameters to unify the data dimensions of environmental parameters. This provides high-quality and reliable input data for subsequent scene evaluation, feature extraction and multimodal fusion, which helps to improve the target recognition accuracy of the intelligent telescope.

[0009] Furthermore, based on the real-time environmental parameters, the scene complexity of the target observation scene is evaluated, and the target recognition process is controlled by the smart telescope at the device end or edge end according to the scene complexity. This includes: acquiring the number of targets, environmental noise intensity, light intensity, and user interaction request type in the target observation scene; if the number of targets is less than a preset number threshold, the environmental noise intensity is less than a preset noise threshold, the light intensity is greater than or equal to a preset illuminance threshold, and the user interaction request type is a basic query, then the scene complexity is determined to be a low-complexity scene, and the smart telescope is controlled to execute the target recognition process locally at the device end; if the number of targets is greater than or equal to a preset number threshold, or the environmental noise intensity is greater than or equal to a preset noise threshold, or the light intensity is less than a preset illuminance threshold, or the user interaction request type is a complex query, then the scene complexity is determined to be a high-complexity scene, and the visual image data, the environmental audio data, and the real-time environmental parameters are compressed and transmitted to the edge end through a low-latency wireless communication link, and the smart telescope is controlled to execute the target recognition process at the edge end.

[0010] In the above scheme, scene complexity is evaluated by combining multi-dimensional indicators such as the number of targets, environmental noise intensity, light intensity, and user interaction needs. For low-complexity scenes, the local computing power of the device is directly allocated for recognition. For high-complexity scenes, the data is compressed and transmitted to the edge via a low-latency wireless link for recognition. This allows for flexible matching of optimal computing resources according to the actual needs of the scene. Thus, it satisfies the recognition accuracy requirements of high-complexity scenes while taking into account power consumption control and response speed in low-complexity scenes, which helps to improve the target recognition accuracy of the smart telescope.

[0011] Furthermore, visual features are extracted from the visual image data, and voiceprint features are extracted from the environmental audio data. The fusion weights of the visual features and voiceprint features are adjusted according to the real-time environmental parameters to obtain target weights for visual features and voiceprint features. This includes: inputting the visual image data into a preset target detection network to obtain visual features; performing time-frequency conversion on the environmental audio data to obtain a spectrogram, and inputting the spectrogram into a preset voiceprint feature extraction network to obtain voiceprint features; and inputting the environmental noise intensity and illumination intensity from the real-time environmental parameters as environmental features into a preset weight allocation network to obtain initial weights for voiceprint features and visual features. Initial weights for features; if the ambient noise intensity is greater than or equal to a preset noise threshold, then the initial weight of the voiceprint feature is reduced and the initial weight of the visual feature is increased to obtain the corresponding visual feature target weight and voiceprint feature target weight; if the illumination intensity is less than a preset illuminance threshold or the scene is backlit, then the initial weight of the voiceprint feature is increased and the initial weight of the visual feature is reduced to obtain the corresponding visual feature target weight and voiceprint feature target weight; wherein, by statistically analyzing the proportion of bright pixels and the proportion of dark pixels in the visual image data, if the proportion of bright pixels meets a preset proportion requirement and the contrast between the bright pixels and the dark pixels meets a preset ratio requirement, then it is determined to be a backlit scene.

[0012] In the above scheme, a corresponding feature extraction network is used to extract features from visual image data and environmental audio data. The fusion weights of the two types of features are adaptively adjusted by combining environmental noise intensity and light intensity. This increases the weight of visual features in high-noise environments and increases the weight of voiceprint features in low-light or backlight scenarios. Therefore, it can fully leverage the information advantages of different modalities in their respective scenarios, achieve dynamic complementarity between the two types of features, effectively avoid recognition biases caused by fixed-weight fusion in extreme environments, and improve the target recognition accuracy of the intelligent telescope.

[0013] Furthermore, after the target recognition process ends, the process also includes: collecting historical user observation behavior data and inputting the observation behavior data into a preset user behavior analysis model to generate a user profile; generating personalized navigation information based on the user profile and the target recognition result; and whenever it is detected that the user's observation time for the same target in the target observation scene exceeds a preset guidance trigger time, controlling the smart telescope to initiate a guided interactive inquiry and outputting the personalized navigation information through voice broadcast or augmented reality display.

[0014] In the above scheme, historical observation data of users is collected and input into a preset user behavior analysis model to generate user profiles. Personalized guide information is generated by matching user profiles with target recognition results. At the same time, when the user's observation time on the same target exceeds a threshold, guided interaction is automatically initiated. This can accurately capture the user's observation preferences and match the corresponding guide content, realizing the upgrade from general information push to personalized customization services. It breaks the interactive limitations of the traditional one-way information display of telescopes and helps to improve the user's sense of participation in the observation process and the adaptability to different needs.

[0015] Furthermore, the intelligent telescope control includes a cloud platform. During the target recognition process, the following steps are also included: if the communication link between the device and the edge is disconnected, the intelligent telescope is controlled to switch to the device to perform the target recognition process using a backup model; wherein the backup model is constructed by the cloud platform based on a preset multimodal fusion model; during the target recognition process performed on the device using the backup model, currently acquired visual image data, environmental audio data, real-time environmental parameters, and target recognition results are cached to obtain a cached dataset; when the communication link between the device and the edge is detected to be restored, the cached dataset is synchronized to both the edge and the cloud.

[0016] In the above solution, the communication link status between the device and the edge is monitored. Whenever the communication link between the device and the edge is lost, the system automatically switches to the cloud and performs recognition locally on the device using a pre-trained backup model. During the disconnection, all collected observation data and recognition results are cached and synchronized to the edge and cloud once the communication link is restored. This ensures the system's basic recognition capabilities in outdoor weak network and no-network scenarios, preventing functional failures caused by communication interruptions. Simultaneously, it fully retains all observation data to support subsequent model optimization and knowledge base updates, improving data integrity and the usability of the smart telescope in complex network environments.

[0017] Furthermore, after the target recognition process ends, the method further includes: if the confidence level of the target recognition result does not meet the preset confidence threshold requirement, acquiring the visual image data, environmental audio data, real-time environmental parameters, and positioning information collected at the current moment, and generating a user prompt; responding to the user prompt confirmation command, integrating the visual image data, environmental audio data, real-time environmental parameters, and positioning information collected at the current moment into a data sample set and uploading it to the cloud; fine-tuning the preset multimodal fusion model based on a preset transfer learning algorithm and the data sample set to obtain an optimized multimodal fusion model; pushing the optimized multimodal fusion model to the edge terminal, and synchronizing the optimized multimodal fusion model to the device terminal through the edge terminal.

[0018] In the above scheme, if the confidence level of the target recognition result does not meet the preset confidence threshold, the system automatically collects current visual, audio, environmental parameters, and positioning information and generates a user prompt. Responding to the user's confirmation command, the data is integrated into a data sample set and uploaded to the cloud. After fine-tuning the model based on a transfer learning algorithm, the data is pushed and synchronized to the edge and device. This allows for rapid incremental optimization of the model with a small number of samples. The closed-loop self-evolution helps improve the accuracy of the preset multimodal fusion model, thereby improving the target recognition accuracy of the smart telescope.

[0019] This invention also provides an intelligent telescope control system based on multimodal fusion perception, installed in an intelligent telescope. The intelligent telescope includes a device end and an edge end. The intelligent telescope control system based on multimodal fusion perception includes a data acquisition module, an intelligent telescope control module, and a target recognition module. The data acquisition module is used to acquire visual image data, environmental audio data, and real-time environmental parameters of the target observation scene. The intelligent telescope control module is used to evaluate the scene complexity of the target observation scene based on the real-time environmental parameters, and control the intelligent telescope to perform a target recognition process at the device end or edge end according to the scene complexity. The target recognition module is used to perform the target recognition process: extracting visual features from the visual image data, extracting voiceprint features from the environmental audio data, and adjusting the fusion weights of the visual features and voiceprint features according to the real-time environmental parameters to obtain visual feature target weights and voiceprint feature target weights; performing weighted fusion of the visual features and voiceprint features based on the visual feature target weights and voiceprint feature target weights to obtain multimodal fusion features; inputting the multimodal fusion features into a preset multimodal fusion model, outputting the target recognition result, and ending the target recognition process.

[0020] This invention proposes an intelligent telescope control system based on multimodal fusion perception. It collects visual image data, environmental audio data, and real-time environmental parameters to obtain multi-source data features. Then, it combines the real-time environmental parameters to assess the scene complexity of the target observation scenario, generating different control strategies to control the intelligent telescope to perform the target recognition process on devices or edge devices with varying computing power. During target recognition, the fusion weights of visual and acoustic features are adaptively adjusted based on real-time environmental parameters to obtain weighted fusion features adapted to different environments. Finally, the multimodal fusion features are input into a preset multimodal fusion model to output reliable target recognition results. Thus, by eliminating the information blind spots of a single modality through the complementary input of multi-source data features, dynamically allocating computing power according to the scene to ensure real-time performance and reliability of recognition, and adaptively adjusting the fusion weights of visual and acoustic features according to environmental parameters to avoid recognition deviations caused by fixed weights in extreme environments, the system achieves adaptive adjustment of the multimodal data fusion strategy based on environmental changes, improving the target recognition accuracy of the intelligent telescope.

[0021] Furthermore, the data acquisition module includes a visual image data acquisition unit, an environmental audio data acquisition unit, and a real-time environmental parameter acquisition unit; wherein: the visual image data acquisition unit is used to continuously acquire real-time video streams of the target observation scene, and perform image signal processing on the visual image data to obtain visual image data; the environmental audio data acquisition unit is used to acquire omnidirectional audio signals within a preset observation range, and perform environmental wind noise and electromagnetic interference suppression processing on the environmental audio data to obtain environmental audio data; the real-time environmental parameter acquisition unit is used to periodically acquire light intensity, environmental noise intensity, and temperature and humidity data of the target observation scene, and perform outlier filtering and normalization processing on the light intensity, environmental noise intensity, and temperature and humidity data respectively to obtain real-time environmental parameters.

[0022] Furthermore, the intelligent telescope control module includes a data acquisition unit, a first control unit, and a second control unit. The data acquisition unit is used to acquire the number of targets, environmental noise intensity, light intensity, and user interaction request type in the target observation scene. The first control unit is used to determine the scene complexity as low complexity if the number of targets is less than a preset quantity threshold, the environmental noise intensity is less than a preset noise threshold, the light intensity is greater than or equal to a preset illuminance threshold, and the user interaction request type is a basic query, and then controls the intelligent telescope to perform the target recognition process locally on the device. The second control unit is used to determine the scene complexity as high complexity if the number of targets is greater than or equal to a preset quantity threshold, or the environmental noise intensity is greater than or equal to a preset noise threshold, or the light intensity is less than a preset illuminance threshold, or the user interaction request type is a complex query, and then compresses the visual image data, the environmental audio data, and the real-time environmental parameters and transmits them to the edge via a low-latency wireless communication link, and controls the intelligent telescope to perform the target recognition process at the edge. Attached Figure Description

[0023] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0024] Figure 1 A flowchart illustrating the steps of an intelligent telescope control method based on multimodal fusion sensing, provided in a certain embodiment of the present invention; Figure 2 This is a schematic diagram of the module structure of an intelligent telescope control system based on multimodal fusion sensing, provided for one embodiment of the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.

[0027] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.

[0028] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0029] In the description of the embodiments in this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0030] In the description of the embodiments of this application, the term "multiple" refers to two or more (including two), similarly, "multiple sets" refers to two or more (including two sets), and "multiple pieces" refers to two or more (including two pieces).

[0031] In the description of the embodiments of this application, unless otherwise expressly specified and limited, technical terms such as "installation," "connection," "joining," and "fixing" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. For those skilled in the art, the specific meaning of the above terms in the embodiments of this application can be understood according to the specific circumstances.

[0032] To address the issue of poor target recognition accuracy in existing smart telescopes due to their reliance on pre-built image recognition models and lack of adaptability to complex environments, see [reference needed]. Figure 1 , Figure 1 This is a flowchart illustrating the steps of a smart telescope control method based on multimodal fusion sensing, provided in one embodiment of the present invention. Figure 1 As shown in the figure, this invention proposes a smart telescope control method based on multimodal fusion sensing, applied to a smart telescope, which includes a device end and an edge end, including steps 101 to 103, each step as follows: Step 101: Collect visual image data, environmental audio data, and real-time environmental parameters of the target observation scene; Step 102: Evaluate the scene complexity of the target observation scene based on the real-time environmental parameters, and control the smart telescope to perform the target recognition process at the device end or edge end according to the scene complexity; Step 103, execute the target recognition process: extract visual features from the visual image data, extract voiceprint features from the environmental audio data, and adjust the fusion weights of the visual features and the voiceprint features according to the real-time environmental parameters to obtain visual feature target weights and voiceprint feature target weights; based on the visual feature target weights and voiceprint feature target weights, perform weighted fusion of the visual features and the voiceprint features to obtain multimodal fusion features; input the multimodal fusion features into a preset multimodal fusion model, output the target recognition result, and end the target recognition process.

[0033] One possible implementation is that the smart telescope adopts a three-tier collaborative architecture combining the device, edge, and cloud. In this architecture, the device is responsible for basic data acquisition, local fast response, and low-power computing, ensuring portability and battery life. The device can be represented as an embedded hardware terminal. The edge is responsible for scheduling real-time computing tasks such as multimodal feature fusion and accurate identification of complex targets, balancing real-time performance and computing power requirements. The edge can be represented as an edge server cluster. The cloud is responsible for knowledge base expansion, model iterative training, and global data optimization, supporting the long-term evolution of the system. The cloud can be represented as a distributed server cluster. The device is also connected to the edge via a 5G NR-U low-latency communication link. After the smart telescope is started, it acquires visual image data of the target observation scene through its internal high-definition optical camera, acquires environmental audio data through an omnidirectional noise-canceling microphone, and acquires environmental parameters in real time through a light sensor and a temperature and humidity sensor.

[0034] Then, the collected data are preprocessed to generate standardized input data, including real-time environmental parameters such as light intensity and environmental noise level, combined with the number of targets to be identified obtained from the preliminary visual image detection, to classify the scene complexity. In this embodiment, scene complexity can be divided into high complexity and low complexity. When it is determined to be a low complexity scene, the complete target recognition process is assigned to the device for local execution to reduce communication power consumption and processing latency. When it is determined to be a high complexity scene, the preprocessed visual image data, environmental audio data, and real-time environmental parameters are subjected to feature compression processing and transmitted to the edge end through the 5G NR-U low-latency communication link to execute the target recognition process.

[0035] After entering the target recognition process, different and corresponding feature extraction networks are used to extract corresponding features based on different data. Visual features are extracted from the visual image data, and voiceprint features are extracted from the environmental audio data. Then, the environmental conditions are judged according to the preset noise threshold and preset illumination threshold applied when classifying scene complexity. This allows the weight of visual features to be increased in high-noise environments and the weight of voiceprint features to be increased in low-light or backlight scenes. In this way, the feature fusion weights are dynamically adjusted to generate corresponding visual feature target weights and voiceprint feature target weights. Finally, the visual feature target weights and voiceprint feature target weights are combined to perform weighted fusion of visual features and voiceprint features to generate a unified multimodal fusion feature. The multimodal fusion feature is then input into a preset multimodal fusion model. The preset multimodal fusion model can be a pre-trained and fine-tuned lightweight multimodal fusion model to complete the recognition of target categories and corresponding attributes, output the final target recognition result, and end the target recognition process. In this embodiment, the preset multimodal fusion model is obtained by pre-training on the LAION-400M dataset, then pre-training on more than 100,000 outdoor target samples, and then fine-tuning with more than 50,000 complex scene samples. The outdoor target samples include birds, mammals and landforms, and the complex scene samples include backgrounds such as low light, high noise and backlight.

[0036] This invention proposes a control method for an intelligent telescope based on multimodal fusion perception. It collects visual image data, environmental audio data, and real-time environmental parameters to obtain multi-source data features. Then, it combines the real-time environmental parameters to assess the scene complexity of the target observation scenario, generating different control strategies to control the intelligent telescope to perform the target recognition process on devices or edge devices with varying computing power. During target recognition, the fusion weights of visual and acoustic features are adaptively adjusted based on real-time environmental parameters to obtain weighted fusion features adapted to different environments. Finally, the multimodal fusion features are input into a preset multimodal fusion model to output reliable target recognition results. Thus, by eliminating the information blind spots of a single modality through the complementary input of multi-source data features, dynamically allocating computing power according to the scene to ensure real-time performance and reliability of recognition, and adaptively adjusting the fusion weights of visual and acoustic features according to environmental parameters to avoid recognition deviations caused by fixed weights in extreme environments, the method achieves adaptive adjustment of the multimodal data fusion strategy according to environmental changes, thereby improving the target recognition accuracy of the intelligent telescope.

[0037] In a preferred embodiment, the acquisition of visual image data, environmental audio data, and real-time environmental parameters of the target observation scene includes: continuously acquiring real-time video streams of the target observation scene; performing image signal processing on the visual image data to obtain visual image data; acquiring omnidirectional audio signals within a preset observation range; performing environmental wind noise and electromagnetic interference suppression processing on the environmental audio data to obtain environmental audio data; and acquiring light intensity, environmental noise intensity, and temperature and humidity data of the target observation scene based on a preset period, and performing outlier filtering and normalization processing on the light intensity, environmental noise intensity, and temperature and humidity data respectively to obtain real-time environmental parameters.

[0038] For example, the device of the smart telescope uses a high-definition optical camera with a continuously adjustable focal length of 8-200mm to continuously acquire real-time video streams of the target observation scene at a frame rate of 30FPS as raw visual image data input. Combined with the optical path calibration parameters of the objective lens group, prism group and eyepiece group in the existing telescope optical system, existing image signal processing is performed on each frame of raw video image, including exposure, gain, white balance, noise reduction and sharpening. This can remove image noise and reduce image blur, accurately restore the true color and texture features of the observation scene, and obtain high-definition and detailed visual image data.

[0039] The device also integrates an omnidirectional noise-canceling microphone to collect omnidirectional audio signals within a preset observation range at a sampling rate of 44.1kHz. Through the built-in A-weighted noise suppression algorithm, it achieves a noise suppression effect of no less than 30dB. It adaptively filters and suppresses environmental wind noise and circuit electromagnetic interference signals in the original audio signal, retains the effective frequency band of the target voiceprint signal, and obtains environmental audio data.

[0040] It also uses the light sensor, temperature and humidity sensor and noise sensor on the device to collect light intensity, ambient noise intensity and temperature and humidity data of the target observation scene at a preset collection cycle of once per second. The Kalman filter algorithm is used to filter out the outliers of the collected raw environmental data, remove invalid data caused by sudden jitter and instantaneous interference of the sensor, and then use the existing normalization method to map all environmental parameters to a unified numerical range to obtain standardized real-time environmental parameters. It is worth mentioning that, for environmental parameters, this embodiment of the invention also proposes an adaptive decision-making process based on a deep Q-network, including taking environmental parameters and image quality scores as state inputs, and outputting actions such as HDR enhancement, white balance adjustment, or edge enhancement through a deep Q-network. The image quality score is represented by the PSNR value, and the calculation formula is as follows: ; In the formula, MAX I MAX represents the maximum grayscale value of an image pixel. In this embodiment, the high-definition image has an 8-bit depth. I =255; MSE is the mean square error between the original image and the reference image, and its calculation formula is: ; In the formula, H is the image height; W is the image width; C is the number of image channels; I(i,j,k) is the pixel value of the image to be evaluated, and K(i,j,k) is the corresponding pixel value of the reference image.

[0041] The action output can be interpreted as follows: In low-light scenes, HDR multi-frame synthesis and Gaussian denoising algorithms are automatically activated, for example, by fusing three images with different exposure levels; in backlight scenes, white balance parameters are adaptively adjusted, and multi-frame exposure fusion technology is used to eliminate backlight shadows and overexposed areas; in complex background scenes, the Canny edge detection threshold is dynamically adjusted to enhance target contour features and improve the distinction between the target and the background. In this embodiment, the device-side light sensor collects light intensity in real time. When three consecutive frames of data are all <50 lux, it is determined to be a low-light scene; the pixel ratio of bright and dark parts of the visual image is calculated. When the bright part ratio is >20% and the bright-dark part contrast ratio is >3:1, it is determined to be a backlight scene. Here, the bright part is the pixel area with a gray value ≥200, and the dark part is the pixel area with a gray value ≤50; then, the image edge features are extracted using the existing Sobel operator, the edge pixel density and gradient variance are calculated, and after normalization, >0.7 is determined to be a complex background scene.

[0042] In the above scheme, image signal processing is performed on visual images to optimize imaging quality and visual recognition, environmental wind noise and electromagnetic interference are suppressed on audio signals to increase the effective information ratio of audio signals, and outlier filtering and normalization are performed on environmental parameters to unify the data dimensions of environmental parameters. This provides high-quality and reliable input data for subsequent scene evaluation, feature extraction and multimodal fusion, which helps to improve the target recognition accuracy of the intelligent telescope.

[0043] A preferred embodiment assesses the scene complexity of the target observation scene based on the real-time environmental parameters, and controls the smart telescope to perform a target recognition process on the device or edge according to the scene complexity. This includes: acquiring the number of targets, environmental noise intensity, light intensity, and user interaction request type in the target observation scene; if the number of targets is less than a preset quantity threshold, the environmental noise intensity is less than a preset noise threshold, the light intensity is greater than or equal to a preset illuminance threshold, and the user interaction request type is a basic query, then the scene complexity is determined to be a low-complexity scene, and the smart telescope is controlled to perform the target recognition process locally on the device; if the number of targets is greater than or equal to a preset quantity threshold, or the environmental noise intensity is greater than or equal to a preset noise threshold, or the light intensity is less than a preset illuminance threshold, or the user interaction request type is a complex query, then the scene complexity is determined to be a high-complexity scene, and the visual image data, environmental audio data, and real-time environmental parameters are compressed and transmitted to the edge via a low-latency wireless communication link, and the smart telescope is controlled to perform the target recognition process at the edge.

[0044] For example, a lightweight target detection model deployed on the device is used to quickly perform preliminary detection on visual image data, count the number of targets to be identified in the current observation scene, and extract the current environmental noise intensity and light intensity values ​​from the standardized real-time environmental parameters, and then obtain the type of interaction request initiated by the current user. Then, the four types of parameters are compared with preset thresholds in turn. When the number of targets is less than the preset number threshold, the environmental noise intensity is less than the preset noise threshold, the light intensity is greater than or equal to the preset illuminance threshold, and the user interaction request type is a basic query, the current target observation scene is determined to be a low-complexity scene. At this time, the computing power scheduling module allocates the complete target recognition process to the device for local execution. In this embodiment, the preset number threshold can be set to 3, the preset noise threshold can be set to 60dB, the preset illuminance threshold can be set to 50lux, and the basic query type can be represented as target name and basic attribute query type. When any one of the four parameters satisfies that the number of targets is greater than or equal to a preset threshold, the environmental noise intensity is greater than or equal to a preset noise threshold, the light intensity is less than a preset illuminance threshold, or the user interaction request type is a complex query, the current target observation scene is determined to be a high-complexity scene. At this time, the preprocessed visual image data, environmental audio data, and real-time environmental parameters are first subjected to feature-level compression processing to compress the volume of the feature data to be transmitted to less than 30% of the original volume. Then, the compressed data is transmitted to the edge end through the 5G NR-U low-latency wireless communication link so that the edge end can perform the target recognition process. In this embodiment, the complex query type can be characterized as in-depth popular science and multi-round association query type.

[0045] In the above scheme, scene complexity is evaluated by combining multi-dimensional indicators such as the number of targets, environmental noise intensity, light intensity, and user interaction needs. For low-complexity scenes, the local computing power of the device is directly allocated for recognition. For high-complexity scenes, the data is compressed and transmitted to the edge via a low-latency wireless link for recognition. This allows for flexible matching of optimal computing resources according to the actual needs of the scene. Thus, it satisfies the recognition accuracy requirements of high-complexity scenes while taking into account power consumption control and response speed in low-complexity scenes, which helps to improve the target recognition accuracy of the smart telescope.

[0046] A preferred embodiment involves extracting visual features from the visual image data, extracting voiceprint features from the environmental audio data, and adjusting the fusion weights of the visual features and the voiceprint features according to the real-time environmental parameters to obtain target weights for the visual features and the voiceprint features. This includes: inputting the visual image data into a preset target detection network to obtain visual features; performing time-frequency conversion on the environmental audio data to obtain a spectrogram, and inputting the spectrogram into a preset voiceprint feature extraction network to obtain voiceprint features; and inputting the environmental noise intensity and illumination intensity from the real-time environmental parameters as environmental features into a preset weight allocation network to obtain initial weights for the voiceprint features and the visual features. Initial weights for visual features are determined; if the ambient noise intensity is greater than or equal to a preset noise threshold, the initial weights for voiceprint features are reduced and the initial weights for visual features are increased to obtain the corresponding target weights for visual features and voiceprint features; if the illumination intensity is less than a preset illuminance threshold or the scene is backlit, the initial weights for voiceprint features are increased and the initial weights for visual features are reduced to obtain the corresponding target weights for visual features and voiceprint features; wherein, by statistically analyzing the proportion of bright pixels and the proportion of dark pixels in the visual image data, if the proportion of bright pixels meets a preset proportion requirement and the contrast between bright pixels and dark pixels meets a preset ratio requirement, then it is determined to be a backlit scene.

[0047] For example, the preprocessed visual image data is input into a preset target detection network embedded in a lightweight module (EfficientChannel Attention ECA module). In this embodiment, the embedded lightweight module can be interpreted using an improved YOLOv13 lightweight model. Visual feature extraction is completed while optimizing the number of model parameters and computational efficiency, outputting 512-dimensional visual features, and the target box localization error is controlled within 2%. Subsequently, a short-time Fourier transform is performed on the preprocessed environmental audio data to convert the one-dimensional time-domain audio signal into a two-dimensional spectrogram. The generated spectrogram is then input into a preset voiceprint feature extraction network to complete voiceprint feature extraction, outputting 256-dimensional voiceprint features. In this embodiment, the preset voiceprint feature extraction network can be interpreted using a ResNet34 network.

[0048] Environmental noise intensity and light intensity data are extracted from standardized real-time environmental parameters and integrated into 64-dimensional environmental features. These environmental features are then input into a pre-defined weight allocation network constructed based on a Dynamic Attention Module (DAM) mechanism. The weight coefficients of the 512-dimensional visual features and the 256-dimensional audio features are dynamically calculated using the Sigmoid function. vis and l aud , where l satisfies vis +l aud =1, weighting coefficient l visThe initial weights are represented by voiceprint features, with weight coefficients l. aud Represented as initial weights for visual features; Then, the current ambient noise intensity is compared with a preset noise threshold. If the ambient noise intensity is detected to be greater than or equal to the preset noise threshold, it indicates that the audio signal is highly affected by environmental interference in the current scene and the proportion of effective voiceprint features has decreased. The initial weight of the voiceprint features is automatically reduced and the initial weight of the visual features is correspondingly increased to generate target weights for visual features and voiceprint features that are adapted to high-noise scenes. In this embodiment, the preset noise threshold is set to 60dB. When the ambient noise intensity is ≥60dB, the signal-to-noise ratio of the audio signal will be lower than 15dB, resulting in the loss of more than 20% of the effective information of the audio features. At this time, the audio feature weight is automatically reduced, for example, from the default 0.4 to 0.2, and the visual feature weight is increased, for example, from the default 0.6 to 0.8. The current light intensity is then compared with a preset illuminance threshold. If the light intensity is detected to be less than the preset illuminance threshold or the scene is determined to be backlit, it indicates that the effective feature information of the visual image is reduced and the image quality is degraded in the current scene. The initial weight of the voiceprint feature is automatically increased and the initial weight of the visual feature is correspondingly decreased to generate target weights for visual features and target weights for voiceprint features that are adapted to low light or backlit scenes. In this embodiment, the preset illuminance threshold is set to 500 lux. When the light intensity is < 500 lux, the contrast and detail loss rate of the visual image exceeds 25%. At this time, the visual feature weight is automatically reduced, for example, from the default 0.6 to 0.4, and the audio feature weight is increased, for example, from the default 0.4 to 0.6.

[0049] It is worth mentioning that by statistically analyzing all pixels in the current visual image data, the overall proportion of bright pixels and the overall proportion of dark pixels in the image are calculated separately, and then the grayscale contrast between the bright and dark areas is calculated. When it is detected that the proportion of bright pixels meets the preset proportion requirement and the contrast between bright and dark pixels meets the preset ratio requirement, the current observed scene is determined to be a backlight scene. For example, when the proportion of bright pixels is >20% and the contrast between bright and dark pixels is >3:1, it is determined to be a backlight scene. Here, bright areas are pixel areas with grayscale values ​​≥200, and dark areas are pixel areas with grayscale values ​​≤50.

[0050] In the above scheme, a corresponding feature extraction network is used to extract features from visual image data and environmental audio data. The fusion weights of the two types of features are adaptively adjusted by combining environmental noise intensity and light intensity. This increases the weight of visual features in high-noise environments and increases the weight of voiceprint features in low-light or backlight scenarios. Therefore, it can fully leverage the information advantages of different modalities in their respective scenarios, achieve dynamic complementarity between the two types of features, effectively avoid recognition biases caused by fixed-weight fusion in extreme environments, and improve the target recognition accuracy of the intelligent telescope.

[0051] In a preferred embodiment, after the target recognition process is completed, the method further includes: collecting historical observation behavior data of the user and inputting the observation behavior data into a preset user behavior analysis model to generate a user profile; generating personalized guide information based on the user profile and the target recognition result; and whenever it is detected that the user's observation time for the same observation target in the target observation scene exceeds a preset guidance trigger time, controlling the smart telescope to initiate a guided interactive inquiry and outputting the personalized guide information through voice broadcast or augmented reality display.

[0052] For example, historical observation behavior data of users is continuously collected. This data includes multi-dimensional data such as the species categories frequently observed by the user, the duration of continuous observation of a single target, the frequency of historical voice interactions, and the types of navigation information viewed in the past. The collected full-volume observation behavior data is then input into an existing LSTM-Transformer user behavior analysis model. The model performs multi-dimensional analysis and fitting of the user's observation preferences and interaction habits to generate a user profile matching these habits. Subsequently, the generated user profile is combined with the current target recognition results to match corresponding popular science and geographical information in the knowledge base to generate personalized navigation information. If the user profile indicates a preference for bird observation, the navigation information prioritizes matching in-depth popular science content such as the breeding habits, distribution areas, and protection levels of the corresponding birds. If the user profile indicates a preference for landform observation, the navigation information prioritizes matching content such as the geological formation causes of the corresponding landform and surrounding landmarks, based on the current GPS positioning information.

[0053] Meanwhile, during the user's observation process, the continuous observation time of the user on the same observation target is counted in real time. Whenever the observation time exceeds the preset guidance trigger time, such as 20 seconds, the smart telescope is controlled to initiate a guided interactive inquiry. The subsequent interactive commands of the user are responded to through the omnidirectional microphone and speech recognition function. The speech recognition function is built according to the Conformer architecture with an added dialect adaptation layer, which supports the accurate recognition of Mandarin and multiple mainstream dialects. The accuracy rate of Mandarin scene recognition is ≥90%, the accuracy rate of dialect scene recognition is ≥80%, and the response latency is ≤500ms. It is also proposed to use the existing mature natural language understanding model based on BERT-base to realize multi-turn context association. Finally, the generated personalized tour guide information is output as audio through the voice broadcast module, and the tour guide content is superimposed on the common image plane of the optical imaging through the projection group and OLED display to complete the augmented reality display, i.e., AR display.

[0054] In the above scheme, historical observation data of users is collected and input into a preset user behavior analysis model to generate user profiles. Personalized guide information is generated by matching user profiles with target recognition results. At the same time, when the user's observation time on the same target exceeds a threshold, guided interaction is automatically initiated. This can accurately capture the user's observation preferences and match the corresponding guide content, realizing the upgrade from general information push to personalized customization services. It breaks the interactive limitations of the traditional one-way information display of telescopes and helps to improve the user's sense of participation in the observation process and the adaptability to different needs.

[0055] In a preferred embodiment, the intelligent telescope control includes a cloud platform. During the target recognition process, the control further includes: if the communication link between the device and the edge device is disconnected, the intelligent telescope is switched to the device to perform the target recognition process using a backup model; wherein the backup model is constructed by the cloud platform based on a preset multimodal fusion model; during the target recognition process on the device platform using the backup model, currently acquired visual image data, environmental audio data, real-time environmental parameters, and target recognition results are cached to obtain a cached dataset; when the communication link between the device and the edge device is detected to be restored, the cached dataset is synchronized to both the edge device and the cloud platform.

[0056] For example, the connection status of the 5G NR-U low-latency communication link between the device and the edge can be monitored in real time. When the communication link is disconnected or the cross-end data transmission is interrupted, the offline operation switching mechanism is automatically triggered. The smart telescope can be switched to the local operation mode of the device without manual operation by the user. The compressed backup model is called to perform the complete target recognition process. The backup model is constructed by the cloud based on the preset lightweight multimodal fusion model and compressed and optimized by knowledge distillation technology.

[0057] During target recognition using the backup model on the device side, the main control board will, without affecting the front-end observation and recognition functions, encrypt all visual image data, environmental audio data, real-time environmental parameters, and target recognition results obtained locally within the current period, and then cache them completely on the device side. By continuously monitoring the connection status of the communication link, when the communication link between the device side and the edge end is detected to have been restored to normal, the integrity of the cached dataset locally cached on the device side is first verified. After confirming that the data is not damaged or lost, the cached dataset is first synchronized to the edge end for data processing and feature verification through a low-latency communication link using a silent background transmission method. Then, the edge end synchronizes the complete cached dataset to the distributed server cluster in the cloud, incorporating the sample data collected in the offline stage into the subsequent model fine-tuning and knowledge base optimization process.

[0058] In the above solution, the communication link status between the device and the edge is monitored. Whenever the communication link between the device and the edge is lost, the system automatically switches to the cloud and performs recognition locally on the device using a pre-trained backup model. During the disconnection, all collected observation data and recognition results are cached and synchronized to the edge and cloud once the communication link is restored. This ensures the system's basic recognition capabilities in outdoor weak network and no-network scenarios, preventing functional failures caused by communication interruptions. Simultaneously, it fully retains all observation data to support subsequent model optimization and knowledge base updates, improving data integrity and the usability of the smart telescope in complex network environments.

[0059] In a preferred embodiment, after the target recognition process ends, the method further includes: if the confidence level of the target recognition result does not meet a preset confidence threshold requirement, acquiring the visual image data, environmental audio data, real-time environmental parameters, and positioning information collected at the current moment, and generating a user prompt; responding to the user prompt confirmation command, integrating the visual image data, environmental audio data, real-time environmental parameters, and positioning information collected at the current moment into a data sample set and uploading it to the cloud; fine-tuning the preset multimodal fusion model based on a preset transfer learning algorithm and the data sample set to obtain an optimized multimodal fusion model; pushing the optimized multimodal fusion model to the edge terminal, and synchronizing the optimized multimodal fusion model to the device terminal through the edge terminal.

[0060] For example, the confidence level of the target recognition result is verified. If the confidence level of the target recognition result does not meet the preset confidence threshold requirement, the current target is determined to be an unrecognized target. A sample collection command is automatically triggered, and the visual image data, environmental audio data, real-time environmental parameters and the current device positioning information collected at the current moment are acquired simultaneously. The user is then prompted with a message asking "Do you want to submit a sample to improve the system?" In this embodiment, the preset confidence threshold requirement is less than 50%.

[0061] Upon receiving a confirmation instruction from the user regarding the prompt information, the collected visual image data, environmental audio data, real-time environmental parameters, and current device location information are encrypted and integrated into a complete data sample set. This data is then uploaded to a distributed server cluster in the cloud via a low-latency communication link using silent background transmission. After receiving the data sample set in the cloud, the uploaded sample data is incrementally fine-tuned using a transfer learning algorithm based on the existing lightweight multimodal fusion pre-trained model. Only a small number of valid samples are needed to complete the model adaptation and optimization, eliminating the need for full model training. After fine-tuning, an optimized multimodal fusion model and corresponding target knowledge entries are generated. In this embodiment, a small number of samples can be interpreted as more than or equal to 10 images or more than or equal to 5 audio segments.

[0062] After model optimization is complete, the cloud pushes the optimized multimodal fusion model and corresponding knowledge entries to the edge device via Over-The-Air (OTA) technology. Upon receiving the updated content, the edge device first calibrates its own computing node's model according to the new model weights, and then synchronizes the optimized multimodal fusion model to the corresponding device via a low-latency communication link. Simultaneously, the updated knowledge entries are synchronized to the interactive semantic database between the device and the edge device. This ensures that users can subsequently access relevant science information about the corresponding target.

[0063] In the above scheme, if the confidence level of the target recognition result does not meet the preset confidence threshold, the system automatically collects current visual, audio, environmental parameters, and positioning information and generates a user prompt. Responding to the user's confirmation command, the data is integrated into a data sample set and uploaded to the cloud. After fine-tuning the model based on a transfer learning algorithm, the data is pushed and synchronized to the edge and device. This allows for rapid incremental optimization of the model with a small number of samples. The closed-loop self-evolution helps improve the accuracy of the preset multimodal fusion model, thereby improving the target recognition accuracy of the smart telescope.

[0064] Based on the above method embodiments, corresponding apparatus embodiments are provided; see [link to apparatus embodiments]. Figure 2 , Figure 2 This is a schematic diagram of the module structure of an intelligent telescope control system based on multimodal fusion sensing, provided as an embodiment of the present invention. Figure 2 As shown in the figure, this embodiment of the invention also provides an intelligent telescope control system based on multimodal fusion perception, which is installed in an intelligent telescope. The intelligent telescope includes a device end and an edge end. The intelligent telescope control system based on multimodal fusion perception includes a data acquisition module 201, an intelligent telescope control module 202, and a target recognition module 203. The data acquisition module 201 is used to acquire visual image data, environmental audio data, and real-time environmental parameters of the target observation scene. The intelligent telescope control module 202 is used to evaluate the scene complexity of the target observation scene based on the real-time environmental parameters and control the intelligent telescope according to the scene complexity. The telescope performs a target recognition process at the device end or edge end; the target recognition module 203 is used to perform the target recognition process: extracting visual features from the visual image data, extracting voiceprint features from the environmental audio data, and adjusting the fusion weights of the visual features and the voiceprint features according to the real-time environmental parameters to obtain visual feature target weights and voiceprint feature target weights; based on the visual feature target weights and voiceprint feature target weights, performing weighted fusion of the visual features and the voiceprint features to obtain multimodal fusion features; inputting the multimodal fusion features into a preset multimodal fusion model, outputting the target recognition result, and ending the target recognition process.

[0065] This invention proposes an intelligent telescope control system based on multimodal fusion perception. It collects visual image data, environmental audio data, and real-time environmental parameters to obtain multi-source data features. Then, it combines the real-time environmental parameters to assess the scene complexity of the target observation scenario, generating different control strategies to control the intelligent telescope to perform the target recognition process on devices or edge devices with varying computing power. During target recognition, the fusion weights of visual and acoustic features are adaptively adjusted based on real-time environmental parameters to obtain weighted fusion features adapted to different environments. Finally, the multimodal fusion features are input into a preset multimodal fusion model to output reliable target recognition results. Thus, by eliminating the information blind spots of a single modality through the complementary input of multi-source data features, dynamically allocating computing power according to the scene to ensure real-time performance and reliability of recognition, and adaptively adjusting the fusion weights of visual and acoustic features according to environmental parameters to avoid recognition deviations caused by fixed weights in extreme environments, the system achieves adaptive adjustment of the multimodal data fusion strategy based on environmental changes, improving the target recognition accuracy of the intelligent telescope.

[0066] Furthermore, the data acquisition module 201 includes a visual image data acquisition unit 301, an environmental audio data acquisition unit 302, and a real-time environmental parameter acquisition unit 303; wherein: the visual image data acquisition unit 301 is used to continuously acquire real-time video streams of the target observation scene, and perform image signal processing on the visual image data to obtain visual image data; the environmental audio data acquisition unit 302 is used to acquire omnidirectional audio signals within a preset observation range, and perform environmental wind noise and electromagnetic interference suppression processing on the environmental audio data to obtain environmental audio data; the real-time environmental parameter acquisition unit 303 is used to periodically acquire light intensity, environmental noise intensity, and temperature and humidity data of the target observation scene, and perform outlier filtering and normalization processing on the light intensity, environmental noise intensity, and temperature and humidity data respectively to obtain real-time environmental parameters.

[0067] Furthermore, the intelligent telescope control module 202 includes a data acquisition unit 401, a first control unit 402, and a second control unit 403. The data acquisition unit 401 is used to acquire the number of targets, environmental noise intensity, light intensity, and user interaction request type in the target observation scene. The first control unit 402 is used to determine the scene complexity as low complexity if the number of targets is less than a preset quantity threshold, the environmental noise intensity is less than a preset noise threshold, the light intensity is greater than or equal to a preset illuminance threshold, and the user interaction request type is a basic query, and then controls the intelligent telescope to perform the target recognition process locally on the device. The second control unit 403 is used to determine the scene complexity as high complexity if the number of targets is greater than or equal to a preset quantity threshold, or the environmental noise intensity is greater than or equal to a preset noise threshold, or the light intensity is less than a preset illuminance threshold, or the user interaction request type is a complex query, and then compresses the visual image data, the environmental audio data, and the real-time environmental parameters and transmits them to the edge end through a low-latency wireless communication link, and controls the intelligent telescope to perform the target recognition process at the edge end.

[0068] It is understood that the above-described device embodiments correspond to the method embodiments of the present invention, and can implement the intelligent telescope control method based on multimodal fusion sensing provided by any of the above-described method embodiments of the present invention.

[0069] It should be noted that the device embodiments described above are merely illustrative, and some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can specifically be implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0070] Based on the above embodiment of an intelligent telescope control method based on multimodal fusion sensing, another embodiment of the present invention provides a terminal device, which includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements an intelligent telescope control method based on multimodal fusion sensing according to any embodiment of the present invention.

[0071] For example, in this embodiment, the computer program can be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the terminal device.

[0072] The terminal device may be a desktop computer, laptop, handheld computer, or cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.

[0073] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the terminal device, connecting all parts of the terminal device via various interfaces and lines.

[0074] Based on the above-described method embodiments, another embodiment of the present invention provides a computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to execute the intelligent telescope control method based on multimodal fusion sensing described in any of the above-described method embodiments of the present invention.

[0075] The modules / units integrated in the device / terminal equipment, if implemented as software functional units and sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0076] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A smart telescope control method based on multimodal fusion sensing, characterized in that, The method for controlling an intelligent telescope, which includes a device end and an edge end, is applied to a smart telescope and includes: Collect visual image data, environmental audio data, and real-time environmental parameters of the target observation scene; The scene complexity of the target observation scene is evaluated based on the real-time environmental parameters, and the target recognition process is controlled by the smart telescope at the device end or edge end according to the scene complexity. The target recognition process is as follows: visual features are extracted from the visual image data, voiceprint features are extracted from the environmental audio data, and the fusion weights of the visual features and voiceprint features are adjusted according to the real-time environmental parameters to obtain visual feature target weights and voiceprint feature target weights; based on the visual feature target weights and voiceprint feature target weights, the visual features and voiceprint features are weighted and fused to obtain multimodal fusion features; the multimodal fusion features are input into a preset multimodal fusion model, the target recognition result is output, and the target recognition process ends.

2. The intelligent telescope control method based on multimodal fusion sensing as described in claim 1, characterized in that, The acquisition of visual image data, environmental audio data, and real-time environmental parameters of the target observation scene includes: Continuously acquire real-time video streams of the target observation scene, and perform image signal processing on the visual image data to obtain visual image data; Omnidirectional audio signals are collected within a preset observation range, and environmental audio data is processed to suppress environmental wind noise and electromagnetic interference to obtain environmental audio data. Based on the collection of light intensity, ambient noise intensity, and temperature and humidity data of the target observation scene at a preset period, outlier filtering and normalization are performed on the light intensity, ambient noise intensity, and temperature and humidity data respectively to obtain real-time environmental parameters.

3. The intelligent telescope control method based on multimodal fusion sensing as described in claim 2, characterized in that, The scene complexity of the target observation scene is evaluated based on the real-time environmental parameters, and the target recognition process is controlled by the smart telescope at the device end or edge end according to the scene complexity, including: Acquire the number of targets, environmental noise intensity, light intensity, and user interaction needs in the target observation scene; If the number of targets is less than a preset number threshold, the environmental noise intensity is less than a preset noise threshold, the light intensity is greater than or equal to a preset illuminance threshold, and the user interaction request type is a basic query, then the scene complexity is determined to be a low complexity scene, and the smart telescope is controlled to execute the target recognition process locally on the device. If the number of targets is greater than or equal to a preset number threshold, or the environmental noise intensity is greater than or equal to a preset noise threshold, or the light intensity is less than a preset illuminance threshold, or the user interaction request type is a complex query, then the scene complexity is determined to be a high-complexity scene. The visual image data, the environmental audio data, and the real-time environmental parameters are compressed and transmitted to the edge via a low-latency wireless communication link, and the smart telescope is controlled to perform the target recognition process at the edge.

4. The intelligent telescope control method based on multimodal fusion sensing as described in claim 3, characterized in that, Visual features are extracted from the visual image data, and voiceprint features are extracted from the environmental audio data. The fusion weights of the visual features and voiceprint features are adjusted according to the real-time environmental parameters to obtain target weights for the visual features and voiceprint features, including: The visual image data is input into a preset target detection network to obtain visual features; The environmental audio data is converted from time to frequency to obtain a spectrogram, and the spectrogram is then input into a preset voiceprint feature extraction network to obtain voiceprint features. The environmental noise intensity and light intensity in the real-time environmental parameters are used as environmental features input into a preset weight allocation network to obtain the initial weights of the voiceprint features and the initial weights of the visual features. If the ambient noise intensity is greater than or equal to a preset noise threshold, then the initial weight of the voiceprint feature is reduced and the initial weight of the visual feature is increased to obtain the corresponding visual feature target weight and voiceprint feature target weight. If the light intensity is less than a preset illuminance threshold or the scene is backlit, the initial weight of the voiceprint feature is increased and the initial weight of the visual feature is decreased to obtain the corresponding visual feature target weight and voiceprint feature target weight. Among them, by statistically analyzing the proportion of bright pixels and the proportion of dark pixels in the visual image data, if the proportion of bright pixels meets the preset proportion requirement and the contrast between the bright pixels and the dark pixels meets the preset ratio requirement, then it is determined to be a backlit scene.

5. A smart telescope control method based on multimodal fusion sensing as described in any one of claims 1 to 4, characterized in that, After the target identification process is completed, it also includes: Collect historical user behavior data and input the observed behavior data into a preset user behavior analysis model to generate a user profile; Personalized navigation information is generated based on the user profile and the target recognition results; Whenever it is detected that the user stays and observes the same target in the target observation scene for a longer than the preset guidance trigger time, the smart telescope is controlled to initiate a guided interactive inquiry and output the personalized guide information through voice broadcast or augmented reality display.

6. The intelligent telescope control method based on multimodal fusion sensing as described in claim 1, characterized in that, The intelligent telescope control system includes a cloud-based component, and the target recognition process also includes: If the communication link between the device and the edge is disconnected, the smart telescope is controlled to switch to the device to perform the target recognition process using a backup model; wherein, the backup model is constructed by the cloud based on a preset multimodal fusion model; During the target recognition process performed by the backup model on the device, the currently acquired visual image data, environmental audio data, real-time environmental parameters and target recognition results are cached to obtain a cached dataset; When the communication link between the device and the edge is restored, the cached dataset is synchronized to the edge and the cloud.

7. The intelligent telescope control method based on multimodal fusion sensing as described in claim 6, characterized in that, After the target identification process is completed, it also includes: If the confidence level of the target recognition result does not meet the preset confidence threshold requirement, the visual image data, environmental audio data, real-time environmental parameters and positioning information collected at the current moment are obtained, and a user prompt is generated; In response to the user's confirmation command, the system integrates the visual image data, environmental audio data, real-time environmental parameters, and positioning information collected at the current moment into a data sample set and uploads it to the cloud. Based on the preset transfer learning algorithm and the data sample set, the preset multimodal fusion model is fine-tuned to obtain an optimized multimodal fusion model; The optimized multimodal fusion model is pushed to the edge terminal, and then synchronized to the device terminal via the edge terminal.

8. A smart telescope control system based on multimodal fusion sensing, characterized in that, An intelligent telescope is configured, comprising a device end and an edge end. The intelligent telescope control system based on multimodal fusion perception includes a data acquisition module, an intelligent telescope control module, and a target recognition module; wherein: The data acquisition module is used to collect visual image data, environmental audio data, and real-time environmental parameters of the target observation scene; The intelligent telescope control module is used to evaluate the scene complexity of the target observation scene based on the real-time environmental parameters, and control the intelligent telescope to perform the target recognition process at the device end or edge end according to the scene complexity. The target recognition module is used to perform the target recognition process: extracting visual features from the visual image data, extracting voiceprint features from the environmental audio data, and adjusting the fusion weights of the visual features and the voiceprint features according to the real-time environmental parameters to obtain visual feature target weights and voiceprint feature target weights; performing weighted fusion of the visual features and the voiceprint features based on the visual feature target weights and voiceprint feature target weights to obtain multimodal fusion features; inputting the multimodal fusion features into a preset multimodal fusion model, outputting the target recognition result, and ending the target recognition process.

9. The intelligent telescope control system based on multimodal fusion sensing as described in claim 8, characterized in that, The data acquisition module includes a visual image data acquisition unit, an environmental audio data acquisition unit, and a real-time environmental parameter acquisition unit; wherein: The visual image data acquisition unit is used to continuously acquire real-time video streams of the target observation scene, and to perform image signal processing on the visual image data to obtain visual image data. The environmental audio data acquisition unit is used to collect omnidirectional audio signals within a preset observation range, and to perform environmental wind noise and electromagnetic interference suppression processing on the environmental audio data to obtain environmental audio data. The real-time environmental parameter acquisition unit is used to periodically collect data on light intensity, ambient noise intensity, and temperature and humidity of the target observation scene, and to perform outlier filtering and normalization on the light intensity, ambient noise intensity, and temperature and humidity data to obtain real-time environmental parameters.

10. The intelligent telescope control system based on multimodal fusion sensing as described in claim 9, characterized in that, The intelligent telescope control module includes a data acquisition unit, a first control unit, and a second control unit; The data acquisition unit is used to acquire the number of targets, environmental noise intensity, light intensity, and user interaction needs in the target observation scene; The first control unit is configured to determine that the scene complexity is a low-complexity scene if the number of targets is less than a preset number threshold, the ambient noise intensity is less than a preset noise threshold, the light intensity is greater than or equal to a preset illuminance threshold, and the user interaction requirement type is a basic query, and then control the smart telescope to perform the target recognition process locally on the device. The second control unit is used to determine that the scene complexity is a high-complexity scene if the number of targets is greater than or equal to a preset number threshold, or the environmental noise intensity is greater than or equal to a preset noise threshold, or the light intensity is less than a preset illuminance threshold, or the user interaction request type is a complex query. The control unit then compresses the visual image data, the environmental audio data, and the real-time environmental parameters and transmits them to the edge end through a low-latency wireless communication link, thereby controlling the smart telescope to perform the target recognition process at the edge end.

Images