Intelligent interaction method and system based on computer vision

Abstract

The present application belongs to the technical field of intelligent interaction of computer vision, and particularly relates to an intelligent interaction method and system based on computer vision, which comprises the following steps: inputting an image into a network, performing deep convolution by using a deep separable convolution module, flattening each channel and performing one-dimensional discrete cosine transformation, intercepting low-frequency components to construct a frequency domain channel descriptor; performing cross-channel interaction modeling and frequency domain aggregation by one-dimensional convolution, generating channel attention weights by full connection layer transformation and S-type activation, weighting feature maps and performing point-by-point convolution; obtaining a logistic regression score and a probability distribution vector by classification, mapping the score to a feature contribution degree by S-type activation, multiplying the score and the feature contribution degree, and normalizing to generate a scene contribution weight, selecting discrete parameters and fusing continuous parameters of a strategy base vector according to the scene contribution weight, generating a control parameter set, and executing the control parameter set. The present application avoids noise interference, realizes smooth transition of control parameters, and improves control accuracy.

Description

Technical Field

[0001] This invention relates to the field of intelligent interaction technology based on computer vision. More specifically, this invention relates to intelligent interaction methods and systems based on computer vision. Background Technology

[0002] Computer vision-based scene recognition technology has become a core component for achieving seamless interaction and control of intelligent devices. Existing intelligent control systems typically acquire environmental images by deploying visual sensors, and then use models such as deep neural networks to analyze the features of the images to understand the current scene state. After acquiring the scene perception results, the system maps the classification results of visual recognition into specific intelligent device control commands, thereby completing the control process from environmental perception to device response.

[0003] However, in the feature extraction stage, existing vision-based interaction methods often rely solely on global pooling or simple spatial convolution when dealing with scenes with overlapping visual features or variable transition states. They fail to delve into deeper information such as the frequency domain and multi-dimensional cross-channel feature interactions, making the network susceptible to interference from redundant background information when extracting key features. This limits the accuracy and environmental adaptability of multi-scene recognition. In the policy deployment stage, most systems, upon receiving the scene recognition signal, generally employ hard mapping logic, directly invoking the corresponding preset control policy based solely on the scene category with the highest probability in the model output. This control method, based on a single deterministic result, disregards the objective reality of multiple scenes coexisting in real-world environments. When scenes switch, the existing hard mapping logic not only causes abrupt changes in discrete start / stop parameters of the device but also leads to sudden changes in continuous control parameters such as temperature and brightness, resulting in a rigid control policy.

[0004] Referring to patent document CN117173475B, a scene prediction fitting method and system based on machine vision are disclosed. This patent document proposes using a machine vision front-end to obtain a scene image containing targets, employing a convolutional neural network to extract scene feature maps and vectorize them, and outputting the recognition type of each target in the scene image in parallel; then inputting this information into a response control module to obtain the response control quantity for the controlled target.

[0005] While the aforementioned patent documents have improved the efficiency of target recognition and control response based on machine vision in complex scenarios to some extent, and established a basic framework for visual perception to control output, their network feature extraction process still lacks effective screening and attention enhancement mechanisms for channel-level frequency domain features. Under the requirement of model lightweighting, it is difficult to further improve the expressive power of multi-scenario features. The final response control still relies on discrete classification outputs and fixed mapping logic, lacking a mechanism for policy fusion based on the actual probability distribution and feature contribution of multiple scenarios. Therefore, when facing complex mixed scenarios, it is impossible to achieve smooth transition and intelligent reorganization of control parameters, restricting the finesse of intelligent device control and the overall intelligent interactive experience for users. Summary of the Invention

[0006] To address the technical problems in the background art, such as the susceptibility of image recognition networks to redundant information interference in feature extraction and the sudden changes in environmental state caused by the use of hard mapping logic in device control strategies, this invention provides solutions in the following aspects.

[0007] In a first aspect, the present invention provides a computer vision-based intelligent interaction method, comprising:

[0008] S1. Input the acquired scene image data to be identified into a deep neural network, wherein the deep neural network includes a standard two-dimensional convolutional input layer, a backbone network composed of multiple depthwise separable convolutional modules stacked in series, and a classification output layer. Each depthwise convolutional layer and pointwise convolutional layer in the depthwise separable convolutional module is followed by a batch normalization layer and a nonlinear activation function layer, respectively. The depthwise separable convolutional module in the backbone network is used to perform depthwise convolution on the input feature map, and the generated depthwise convolutional feature map is flattened in the spatial dimension and then subjected to a one-dimensional discrete cosine transform. A preset number of low-frequency components are extracted to construct a frequency domain channel descriptor. S2. The frequency domain channel descriptor is modeled across channels using a one-dimensional convolutional layer and aggregated in the frequency domain dimension. The dimension transformation and S-shaped activation mapping are performed sequentially through a fully connected layer to generate attention weights for each channel. The attention weights are then multiplied element-wise with the deep convolutional feature map and pointwise convolution is performed to output the feature map. S3. The feature map is classified to obtain the logistic regression score and the probability distribution vector of each scene category. The logistic regression score is converted into a feature contribution vector by sigmoid activation mapping. The feature contribution vector is multiplied element-wise with the probability distribution vector of the scene category and normalized to generate scene contribution weights. Based on the scene contribution weights, discrete control parameters are selected and continuous control parameter strategies are fused on the preset single scene control strategy basis vector to generate the target intelligent device control strategy parameter set and load it for execution.

[0009] This invention extracts low-frequency components using one-dimensional discrete cosine transform, filters noise while retaining core frequency domain features, and calculates channel attention weights using one-dimensional convolution and fully connected layers to dynamically strengthen key feature channels, thereby improving the expressive power of features in complex scenarios. In the control phase, logistic regression scores are converted into feature contribution values, and scenario contribution weights are generated by combining probability distributions, abandoning conventional hard mapping logic. Based on these weights, discrete parameters are extracted, and continuous control parameters are weighted and fused to truly reflect the coexistence of multiple scenarios, achieving smooth transition and reorganization of equipment control parameters and improving the precision of intelligent control.

[0010] Preferably, the step of flattening each channel of the generated deep convolutional feature map in the spatial dimension and performing a one-dimensional discrete cosine transform to extract a preset number of low-frequency components to construct a frequency domain channel descriptor includes: flattening each channel of the deep convolutional feature map into a one-dimensional sequence in row-major order and independently performing the one-dimensional discrete cosine transform to generate a frequency domain coefficient sequence; and extracting the first K low-frequency coefficients of the frequency domain coefficient sequence in ascending order of frequency and concatenating them into a vector of length K as the frequency domain channel descriptor for the corresponding channel; wherein K is a preset integer greater than 0.

[0011] It can avoid the loss of high-frequency details caused by traditional global average pooling, and retain the frequency domain information inside the channel while reducing the number of computational parameters of the network model.

[0012] Preferably, the step of using a one-dimensional convolutional layer to perform cross-channel interactive modeling of the frequency domain channel descriptors and aggregating them in the frequency domain dimension includes: concatenating the frequency domain channel descriptors of all channels to form a descriptor matrix; transposing the descriptor matrix, performing sliding window convolution in the channel dimension using the one-dimensional convolutional layer, and applying a flattening operation in the frequency domain dimension to perform dimensional concatenation and aggregation to obtain a one-dimensional channel feature vector.

[0013] This invention obtains the local feature dependencies between adjacent channels by performing sliding window convolution in the channel dimension, and performs dimension splicing and aggregation by applying flattening operation in the frequency domain dimension, thereby realizing cross-channel information interaction modeling with low computational complexity.

[0014] Preferably, the fully connected layer includes a first fully connected layer and a second fully connected layer. The step of sequentially performing dimensionality transformation and sigmoid activation mapping through the fully connected layers to generate attention weights for each channel includes: inputting the aggregated one-dimensional channel feature vector into the first fully connected layer to reduce the channel dimension with a preset compression rate to obtain dimensionality-reduced features; inputting the dimensionality-reduced features into the second fully connected layer to restore the channel dimension to the same number of channels as the depthwise convolutional feature map; and applying the sigmoid activation mapping to the feature vector after dimensionality restoration to generate attention weights for each channel.

[0015] This invention obtains the weights of each channel by combining a channel dimension compression and restoration processing mechanism with nonlinear mapping, effectively strengthening the key channels containing real scene features, while autonomously suppressing redundant and invalid features.

[0016] Preferably, the logistic regression score for each scene category is obtained by classifying the feature map, including: inputting the feature map into a global average pooling layer to generate a global feature vector by performing average pooling in the spatial dimension; inputting the global feature vector into a fully connected classifier and performing a linear transformation through the weight matrix of the fully connected classifier to obtain the logistic regression score for each scene category.

[0017] This invention transforms high-dimensional spatial features into low-dimensional category score representations through average pooling in the spatial dimension and linear transformation of a fully connected classifier, providing accurate data support for subsequent weight distribution calculations.

[0018] Preferably, the scene category probability distribution vector is obtained by applying the Softmax function to the logistic regression scores of each scene category for probability normalization.

[0019] This transforms the values ​​of each scene category into a probability distribution that sums to 1, allowing for an intuitive and accurate acquisition of the specific probability parameters of the current image to be identified belonging to each scene dimension.

[0020] Preferably, generating scene contribution weights by multiplying the feature contribution vector element-by-element with the scene category probability distribution vector and normalizing the result includes: performing a Hadamard product operation on the feature contribution vector of each scene with the scene category probability distribution vector to obtain a modulation vector; summing all elements in the modulation vector; and dividing each element in the modulation vector by the summation value to generate the scene contribution weights.

[0021] The absolute feature activation intensity and relative probability distribution of the scene are modulated and combined through the Hadamard product, and then calculated through linear normalization to ensure that the final generated weight values ​​truly map the proportion of different scenes in the actual physical space.

[0022] Preferably, the discrete control parameter selection for the preset single-scene control strategy basis vector based on the scene contribution weight includes: pre-constructing a strategy matrix composed of multiple single-scene control strategy basis vectors; selecting the single-scene control strategy basis vector corresponding to the maximum value of the scene contribution weight from the strategy matrix; and extracting the discrete control parameter setting value from the selected single-scene control strategy basis vector as the execution state.

[0023] Discrete states are selected based on the scenario category with the highest weight to ensure that the start / stop status of smart devices matches the dominant scenario environment.

[0024] Preferably, the continuous control parameter strategy fusion to generate a target intelligent device control strategy parameter set includes: selecting candidate basis vectors from the strategy matrix whose discrete control parameter setpoints are the same as the execution state; extracting the scene contribution weights corresponding to each candidate basis vector and re-normalizing them to generate combination coefficients; using the combination coefficients to perform linear weighted summation on the corresponding continuous control parameters in each candidate basis vector, and combining the execution state to generate the target intelligent device control strategy parameter set.

[0025] While ensuring the consistency of discrete execution states, the scenario contribution weights are renormalized to generate combination coefficients, and the continuous control parameters are linearly weighted and summed to accurately output the weighted average setpoint that matches the mixed physical states of multiple scenarios.

[0026] Secondly, the present invention provides a computer vision-based intelligent interaction system, including a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the above-mentioned computer vision-based intelligent interaction method is implemented.

[0027] By adopting the above technical solution, the computer vision-based intelligent interaction method is generated into a computer program and stored in a memory for loading and execution by a processor. This allows for the creation of a terminal device based on the memory and processor, making it convenient to use.

[0028] The beneficial effects of this invention are as follows: This invention transforms the pixel intensity distribution in the spatial domain into the energy distribution in the frequency domain. It constructs frequency domain channel descriptors by extracting low-frequency components, replacing the conventional global average pooling process. Combined with one-dimensional local cross-channel interactive modeling and dimensional processing of two-level fully connected layers, this neural network can accurately calculate and allocate attention weights for each channel. This guides depthwise separable convolutions to effectively focus on core scene features with high discriminative power. While maintaining low computational complexity, it filters out interference from redundant background noise, improving scene recognition accuracy and multi-dimensional feature expression capabilities in variable environments.

[0029] Furthermore, by nonlinearly mapping the logistic regression score of the visual classification model to the feature contribution and jointly normalizing it with the scene category distribution probability, a scene contribution weight index is constructed. The control system uses this weight to filter the discrete control parameters of the device and performs weighted fusion and summation on the continuous control parameters under the same mode. This achieves a smooth transition and reorganization of the operating parameters of the intelligent device in complex mixed scenes, effectively avoiding sudden changes in temperature and brightness caused by hard switching of single scene strategies, and improving the precision level and environmental adaptability of hardware device control parameter settings under multi-scene perception. Attached Figure Description

[0030] Figure 1 A flowchart of a computer vision-based intelligent interaction method; Figure 2 This is a diagram illustrating channel attention weights. Figure 3 This is a diagram illustrating scene scores and probabilities. Detailed Implementation

[0031] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0032] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0033] This invention discloses an intelligent interaction method based on computer vision, referring to... Figure 1 This includes steps S1-S3: S1. Image preprocessing and frequency domain feature extraction.

[0034] In one embodiment, the acquired scene image data to be identified is input into a deep neural network. The deep separable convolution module in the deep neural network performs deep convolution on the input feature map. The generated deep convolution feature map is then flattened in space and subjected to a one-dimensional discrete cosine transform. A preset number of low-frequency components are extracted to construct frequency domain channel descriptors. A video stream is acquired and extracted frame by frame, or the scene image data to be identified is read from the local hard drive and converted into a three-dimensional multidimensional array. A resolution adjustment operation is performed sequentially to adjust the resolution of the scene image data to 224×224 pixels. A tensor transformation operation is performed to convert the scene image data to a tensor type, and a normalization function is called to perform normalization based on a preset mean and standard deviation.

[0035] The preprocessed image tensor is input into a deep neural network built on a deep learning framework. The deep neural network includes a standard 2D convolutional input layer for receiving and initially feature-mapping the image data of the scene to be identified; a backbone network composed of depthwise separable convolutional modules; and a classification output layer for classifying the final feature maps output by the backbone network to output logistic regression scores for each scene category. The backbone network is composed of multiple depthwise separable convolutional modules stacked sequentially, extracting multi-scale features by progressively reducing the spatial size of the feature maps and progressively increasing the number of feature map channels. Between adjacent depthwise separable convolutional modules, the spatial size is reduced by setting the stride of one of the depthwise convolutional modules to a value greater than 1; for adjacent modules with the same spatial size, skip connections are introduced to form a residual structure. Within each depthwise separable convolutional module, a batch normalization layer and a non-linear activation function layer follow the depthwise convolutional layer and the pointwise convolutional layer, respectively.

[0036] In the specific implementation of this embodiment, for the depth convolutional layer in the depth separable convolutional module, the convolutional layer is instantiated using a two-dimensional convolutional class. During initialization, the input channel parameters are configured to be equal to the output channel parameters, and the grouping parameters are set to the same value as the number of input channels. This allows each input channel to be assigned an independent spatial convolutional kernel to perform a sliding window dot product accumulation operation on the input feature map channel by channel, and outputs a depth convolutional feature map after each channel has independently extracted features.

[0037] After flattening each channel of the generated deep convolutional feature map in the spatial dimension, a one-dimensional discrete cosine transform is performed. A preset number of low-frequency components are extracted to construct a frequency domain channel descriptor. Specifically, the starting dimension parameter is set to 2. The four-dimensional deep convolutional feature map with shape of batch size, number of channels, height, and width is flattened in the height and width spatial dimensions and reconstructed into a three-dimensional tensor with shape of batch size, number of channels, and total number of pixels.

[0038] The channels of the deep convolutional feature map are flattened into a one-dimensional sequence in row-major order. Assuming the size of the input feature map is the number of channels multiplied by the height multiplied by the width, let's set the number of channels to 64, the height to 56, and the width to 56, resulting in a deep convolutional feature map size of 64×56×56. Each 56×56 two-dimensional spatial matrix of the 64 channels is flattened in row-major order into a one-dimensional feature sequence with a length equal to the height multiplied by the width, i.e., a length of 3136.

[0039] A one-dimensional discrete cosine transform (DCT) is independently performed on the one-dimensional sequence of each channel to obtain the frequency domain coefficient sequence of each channel. A fast DCT algorithm is used to independently perform DCT on 64 one-dimensional feature sequences of length 3136, with the normalization parameter set to orthogonality. A one-dimensional second-kind orthogonal DCT is then performed to transform the pixel intensity distribution in the spatial domain into an energy distribution in the frequency domain, generating a one-dimensional frequency domain sequence containing 3136 frequency coefficients. The first K low-frequency coefficients of each frequency domain coefficient sequence are extracted in ascending order of frequency, where K is a preset integer greater than 0. The preset integer K is limited to a range of 8 to 32. In this embodiment, the preset integer K is set to 16, meaning that 16 low-frequency coefficients with indices from 0 to 15 are extracted from the frequency domain sequence. Index 0 corresponds to the DC component, and indices 1 to 15 correspond to the low-frequency AC components.

[0040] The extracted K low-frequency coefficients are concatenated into a vector of length K, serving as the frequency domain channel descriptor for the corresponding channel. After extraction, the 16 low-frequency coefficients are arranged in ascending order of frequency, and high-frequency noise components are removed. They are then concatenated into a feature vector of length 16, forming the frequency domain channel descriptor for a single channel. The 16-dimensional feature vectors extracted from each of the 64 channels are merged along the channel dimension to construct a 64×16 channel descriptor matrix. A frequency domain truncation-based method is used to avoid the loss of high-frequency details caused by traditional global average pooling, and to preserve channel frequency domain information while reducing the number of computational parameters.

[0041] S2, Channel Attention Weight Generation and Feature Recalibration.

[0042] In one embodiment, a one-dimensional convolutional layer is used to model cross-channel interactions of frequency domain channel descriptors and aggregate them in the frequency domain dimension. Then, fully connected layers are used for dimensionality transformation and sigmoid activation mapping to generate attention weights for each channel. These attention weights are then multiplied element-wise with the depthwise convolutional feature map, followed by pointwise convolution to output the feature map. The frequency domain channel descriptors of all channels are concatenated to form a channel descriptor matrix. The 64×16 channel descriptor matrix generated in the previous step is transposed to convert it to a 16×64 matrix, making the channel dimension the direction of the feature sequence.

[0043] This method utilizes a one-dimensional convolutional layer to model cross-channel interactions of frequency domain channel descriptors and aggregates them along the frequency domain dimension. Specifically, it involves transposing the channel descriptor matrix, then performing sliding window convolutions along the channel dimension using a one-dimensional convolutional layer with a preset kernel size to obtain local feature dependencies between adjacent channels. The preset kernel size is limited to an odd number between 3 and 7. In this embodiment, the preset kernel size is configured as 3, the sliding stride is set to 1, and an edge zero-padding algorithm is used to maintain a 64-dimensional channel count. The output matrix of the one-dimensional convolutional layer then has a dimension of 16×64. A flattening operation is applied along the 16-dimensional frequency domain feature dimension to concatenate and aggregate the frequency domain features that have undergone cross-channel interaction, resulting in a 16×16-dimensional channel feature vector.

[0044] The system sequentially performs dimensionality transformation and S-shaped activation mapping through fully connected layers to generate attention weights for each channel. Specifically, this involves inputting the aggregated 64-dimensional one-dimensional channel feature vector into a first fully connected layer with a linear rectified activation function to reduce the channel dimension at a preset compression rate, resulting in dimensionality-reduced features. The preset compression rate can be 4, 8, or 16. In this embodiment, a preset compression rate of 8 is set. Dividing the number of channels by the preset compression rate reduces the channel dimension to the target low-dimensional state, generating 8-dimensional dimensionality-reduced features. The generated 8-dimensional dimensionality-reduced features are then input into a second fully connected layer to restore the channel dimension of the dimensionality-reduced features to 64 dimensions, the same number of channels as the original depthwise convolutional feature map.

[0045] The attention weights for each channel are generated by applying a sigmoid activation map to the feature vector after dimensionality restoration. Specifically, the steps are as follows: A sigmoid non-linear activation function is applied element-wise to each element of the restored 64-dimensional channel feature vector for feature mapping. The operational formula for the sigmoid non-linear activation function is:

[0046] in, This represents the output weight values ​​of the sigmoid nonlinear activation function. The input channel feature vector element is obtained through forward computation of the second fully connected layer. In this embodiment, its value is preset to be a real number representing the feature intensity. This represents a natural constant, which in this embodiment is preset to a dimensionless mathematical constant of 2.71828. Mapping all element values ​​to the interval between 0 and 1 generates a vector of 64 real values, corresponding to the attention weights of the original 64 feature channels. This vector will be used to perform a Hadamard product operation with the channels of the original depthwise convolutional feature map. See also... Figure 2 As shown in the figure, the weight values ​​of channels 0 to 10 and 35 to 40 are relatively high, indicating that the scene features contained in these channels have been enhanced by the model; the weight values ​​of channels 20 to 30 and 50 to 55 are relatively low, indicating that redundant and invalid features have been automatically suppressed.

[0047] The attention weights are element-wise multiplied with the depthwise convolutional feature map, and then pointwise convolution is performed to output the feature map. The specific steps are as follows: The tensor dimensionality transformation method is called to reshape and expand the one-dimensional tensor containing the attention weights of each channel into a four-dimensional tensor with batch size, number of channels, and values ​​of 1 and 1, to match the rules of the tensor broadcasting mechanism. The tensor multiplication function is called to perform a scalar scaling operation, element-wise multiplication of the expanded attention weight tensor with the four-dimensional depthwise convolutional feature map along the corresponding channel positions, and a Hadamard product operation is performed to complete the channel feature recalibration process. A pointwise convolutional layer is constructed on the recalibrated feature map. The pointwise convolutional kernel size is set to 1 and the grouping parameter to 1 in the initialization parameters. The pointwise convolutional layer performs a cross-channel linear combination on the recalibrated feature map, performing dimensionality scaling and cross-channel feature fusion, and finally outputs the feature map processed by the current depthwise separable convolutional module.

[0048] S3, scene weight calculation and control strategy integration and execution.

[0049] In one embodiment, the feature map is classified to obtain the logistic regression score and the probability distribution vector of each scene category. The logistic regression score is then processed by sigmoid activation mapping to generate a feature contribution vector. The feature contribution is multiplied element-wise with the probability distribution vector and normalized to generate scene contribution weights. Based on the scene contribution weights, discrete control parameters are selected and continuous control parameter strategies are fused on the preset single scene control strategy basis vector to generate the target intelligent device control strategy parameter set and load it for execution.

[0050] The feature maps are classified to obtain logistic regression scores for each scene category. This process includes: inputting the feature map output from the last depthwise separable convolutional module in the deep neural network into a global average pooling layer. The size of the feature map is preset to 512 channels multiplied by a height of 7 and a width of 7. In the two-dimensional space, average pooling is performed on the 49 feature values ​​in each channel of the feature map to generate a one-dimensional global feature vector of size 512×1. The global feature vector is then input into a fully connected classifier with no bias term and a preset weight matrix size of 5×512. A linear transformation from 512-dimensional to 5-dimensional classification space is achieved through matrix inner product operations on the weight matrix of the fully connected classifier, yielding the logistic regression score for each scene category.

[0051] The total number of preset scene categories is set to 5, including office scene, movie-watching scene, sleep scene, meeting scene and away-from-home scene. The output is a one-dimensional tensor containing 5 real values. The preset logistic regression score numerical sequence is 2.45, -0.32, 1.15, 0.88 and -1.02 respectively.

[0052] Furthermore, the scene category probability distribution vector is obtained by applying the Softmax function to the logistic regression scores of each scene category for probability normalization. The natural constant exponent is calculated for each of the five logistic regression scores, and the resulting exponent value for each scene category is divided by the sum of the exponent values ​​for all scene categories, ensuring that the probability value for each scene category is between 0 and 1 and the sum of all probability values ​​equals 1. For example, after the aforementioned scores undergo normalized exponent transformation, a 1×5 scene category probability distribution vector of 0.605, 0.038, 0.165, 0.126, and 0.066 is output, representing the probability that the current environment image belongs to each scene. This vector serves as the probability distribution vector for multi-scene classification and is used for subsequent calculation of control strategy parameters.

[0053] Furthermore, the feature contribution is multiplied element-wise by the probability distribution vector and normalized to generate scene contribution weights. This includes: independently applying the sigmoid nonlinear activation function to the logistic regression score of each scene category to perform nonlinear mapping. The formula for the sigmoid nonlinear activation function is:

[0054] in, Indicates the contribution of features. The logistic regression score is obtained through forward computation using the matrix inner product operation of a fully connected classifier. In this embodiment, the preset logistic regression score value is a real number representing the scene activation intensity. The natural constant is obtained through the standard mathematical library of the system programming language. In this embodiment, the preset value of the natural constant is a dimensionless constant of 2.71828.

[0055] After element-by-element calculation using the formula, the above 5 logistic regression scores are converted into feature contribution values, with the preset feature contribution value sequence being 0.920, 0.42, 0.759, 0.706, and 0.265 respectively.

[0056] The vector formed by the feature contribution values ​​of each scene category is multiplied by the scene category probability distribution vector using a Hadamard product. Element-wise scalar multiplication is then performed at the same category index positions to obtain a one-dimensional modulation vector. The preset numerical sequence of the one-dimensional modulation vector is 0.556, 0.016, 0.125, 0.089, and 0.017. All elements in the one-dimensional modulation vector are summed to obtain a total value, with a preset total value of 0.803. Each of the five elements in the one-dimensional modulation vector is divided by the total value of 0.803 to obtain the normalized scene contribution weights. The preset numerical sequence of scene contribution weights is 0.692, 0.020, 0.156, 0.111, and 0.021. These scene contribution weights will be used to determine the fusion ratio of the basis vectors for different single-scene control strategies. (See also...) Figure 3As shown, the original score and probability of the office scenario are the highest among the five scenarios, while the original scores of the movie-watching and away-from-home scenarios are negative, which matches the trend of the example data.

[0057] Because the Softmax function has a competitive characteristic of forced normalization, when multiple features overlap in complex mixed scenes, a single Softmax mapping can suppress the probability distribution weights of scenes with significant secondary features, causing the classification results to deviate from the physical reality of multiple scenes coexisting. In contrast, the sigmoid activation mapping can independently evaluate the absolute feature activation intensity of each scene category, unaffected by the scores of other scene categories. This application uses the feature contribution output of the sigmoid activation mapping as a modulator, performing Hadamard product modulation with the relative probability distribution vector generated by Softmax, effectively counteracting the weight suppression phenomenon caused by a single Softmax. This operational mechanism ensures that the final generated scene contribution weights balance the rationality of the global relative distribution with the local absolute activation intensity, avoiding the loss of significant secondary features at the information theory level, thus providing data support with rigorous derivation logic for the smooth fusion of subsequent continuous control parameters.

[0058] Further, discrete control parameters are selected for the preset single-scene control strategy basis vector based on scene contribution weights. This includes: pre-constructing a strategy matrix in the system database, with the matrix consisting of 4 rows and 5 columns. The 5 columns correspond to the 5 preset scene categories mentioned above, and the 4 rows correspond to the control parameters of the 4 target smart devices. For the discrete control parameters in the strategy matrix, the preset main light switch state parameter is set to 0 for off and 1 for on; the preset color temperature scene mode parameter is set to 1 for 3000K warm light and 2 for 6000K cool light. The single-scene control strategy basis vector corresponding to the maximum scene contribution weight is selected from the pre-constructed strategy matrix. The office scene corresponds to a maximum weight of 0.692, so all discrete control parameter settings in the single-scene control strategy basis vector corresponding to the office scene are selected. The selected discrete control parameter settings in the single-scene control strategy basis vector are extracted as execution states, with the preset main light switch execution state set to 1 and the preset color temperature mode execution state set to 2.

[0059] Furthermore, the continuous control parameter strategy fusion generates a target intelligent device control strategy parameter set, including: for continuous control parameters in the strategy matrix, the preset light brightness percentage parameter range is limited to 0% to 100%; the preset air conditioning set temperature parameter range is limited to 16℃ to 30℃. Candidate basis vectors with discrete control parameter settings and execution states are selected from the strategy matrix. The single-scene control strategy basis vectors of the remaining four scenarios in the strategy matrix are traversed, and the meeting scenario with the main light set to 1 and color temperature mode set to 2 is selected as a candidate basis vector. The movie-watching scenario, sleep scenario, and leaving-home scenario are eliminated because the discrete control parameters are inconsistent with the execution state.

[0060] If no other candidate basis vectors with the same discrete control parameter setpoints and execution states are found from the strategy matrix, the continuous control parameter strategy fusion is stopped, and the continuous control parameters in the single scenario control strategy basis vector corresponding to the maximum scenario contribution weight are directly used as the target intelligent device control strategy parameter set.

[0061] The scene contribution weights corresponding to each candidate basis vector are extracted and re-normalized to generate combination coefficients. The retained office scene weight (0.692) and meeting scene weight (0.111) are extracted and re-normalized, generating a combination coefficient of 0.862 for the office scene and 0.138 for the meeting scene. The combination coefficients are then used to perform a linear weighted summation of the corresponding continuous control parameters in each candidate basis vector. The default brightness for the office scene is preset to 80%, and the default brightness for the meeting scene to 60%. The combined coefficients are then used to perform a linear weighted summation of these two continuous control parameter values, yielding a result of 77.24%. The result is rounded to the nearest integer according to a preset precision, resulting in a target brightness setting of 77%. The target air conditioning temperature setting is preset to 25℃. The extracted execution status and the fused continuous control parameters are then combined to generate a target intelligent device control strategy parameter set.

[0062] Furthermore, the target smart device control strategy parameter set is serialized into a string format data payload and published for transmission to the target smart device for execution. Loading the target smart device control strategy parameter set into the target device for execution specifically includes: importing the `json` module from the Python standard library, calling the `dumps` method to serialize the target smart device control strategy parameter set into a UTF-8 encoded JSON string format data payload; instantiating an MQTT client object, calling the `connect` method to connect to the MQTT broker server deployed on the local area network or in the cloud, and publishing the data payload containing the parameter set as message content by calling the `publish` method to specify the topic subscribed to by the target smart device or home gateway; after receiving the message, the wireless communication module in the target device's microcontroller calls the parsing function to restore the strategy parameters, drives the general-purpose input / output pin level changes to control the relay on / off state, or outputs a pulse width modulation signal through the digital-to-analog converter interface to change the motor speed or bulb brightness, thus completing the smart device state adjustment and control strategy execution corresponding to the multi-scene recognition results.

[0063] Four network models were constructed for comparative analysis. The baseline model uses traditional global average pooling to extract channel features, generates attention weights using fully connected layers, and directly outputs a single-scene control strategy based on the maximum classification probability. Ablation Model 1, based on the baseline model, replaces feature extraction with performing a one-dimensional discrete cosine transform to truncate low-frequency components and obtain frequency domain channel descriptors, and performs local cross-channel interaction modeling through one-dimensional convolution. Ablation Model 2, based on the baseline model, adds a module that maps logistic regression scores to feature contribution and generates scene contribution weights together with the probability distribution, and adopts a fusion strategy of linear combination of continuous control parameters of the same pattern. The complete model includes both the frequency domain channel feature extraction module and the scene contribution weight control parameter fusion mechanism. Experiments used a smart home dataset containing 100,000 multi-scene samples, and the evaluation metrics were multi-scene classification accuracy and mean square error of device control continuous parameters.

[0064] The baseline model achieved a scene classification accuracy of 83.5% and a mean squared error (MSE) of 4.58 for the continuous parameters of device control. After incorporating a frequency domain channel feature extraction, ablation model one improved scene classification accuracy to 87.2% and reduced the MSE of the continuous parameters of device control to 3.75. Ablation model two, due to its parameter strategy fusion mechanism, only slightly increased scene classification accuracy to 84.1%, but reduced the MSE of the continuous parameters of device control to 2.36. The complete model, through the combined effect of both models, achieved the highest scene classification accuracy of 89.8% and the lowest MSE of 1.45 for the continuous parameters of device control.

[0065] Before performing the above steps, the deep neural network is pre-trained using an image dataset containing various known scene category labels. During training, the cross-entropy loss function is used to calculate the classification error between the output scene category probability distribution vector and the true label. The Adam optimizer is then used to iteratively update the network weight parameters of the deep separable convolutional module, one-dimensional convolutional layer, and fully connected layer through backpropagation until the loss function converges, thus obtaining the trained model.

[0066] The present invention also discloses a computer vision-based intelligent interaction system, including a processor and a memory. The memory stores computer program instructions, and when the computer program instructions are executed by the processor, the computer vision-based intelligent interaction method according to the present invention is implemented.

[0067] The aforementioned computer vision-based intelligent interactive system also includes other components well known to those skilled in the art, such as communication buses and communication interfaces. Their settings and functions are known in the art and will not be described in detail here.

[0068] In the description of this specification, "multiple" or "several" means at least two, such as two, three or more, unless otherwise expressly and specifically defined.

Claims

1. A computer vision-based intelligent interaction method, characterized in that, include: S1. Input the acquired scene image data to be identified into a deep neural network, wherein the deep neural network includes a standard two-dimensional convolutional input layer, a backbone network composed of multiple depthwise separable convolutional modules stacked in series, and a classification output layer. Each depthwise convolutional layer and pointwise convolutional layer in the depthwise separable convolutional module is followed by a batch normalization layer and a nonlinear activation function layer, respectively. The depthwise separable convolutional module in the backbone network is used to perform depthwise convolution on the input feature map, and the generated depthwise convolutional feature map is flattened in the spatial dimension and then subjected to a one-dimensional discrete cosine transform. A preset number of low-frequency components are extracted to construct a frequency domain channel descriptor. S2. The frequency domain channel descriptor is modeled across channels using a one-dimensional convolutional layer and aggregated in the frequency domain dimension. The dimension transformation and S-shaped activation mapping are performed sequentially through a fully connected layer to generate attention weights for each channel. The attention weights are then multiplied element-wise with the deep convolutional feature map and pointwise convolution is performed to output the feature map. S3. The feature map is classified to obtain the logistic regression score and scene category probability distribution vector for each scene category. The logistic regression score is then processed using a S-shaped activation mapping to generate a feature contribution vector. The feature contribution vector is element-wise multiplied with the scene category probability distribution vector and normalized to generate scene contribution weights. Based on the scene contribution weights, discrete control parameters are selected and continuous control parameter strategies are fused onto a preset single-scene control strategy basis vector to generate a target intelligent device control strategy parameter set and load it for execution. The discrete control parameter selection based on the scene contribution weights includes: A strategy matrix composed of multiple single-scene control strategy basis vectors is pre-constructed; the single-scene control strategy basis vector corresponding to the maximum value of the scene contribution weight is selected from the strategy matrix; the discrete control parameter setting value in the selected single-scene control strategy basis vector is extracted as the execution state. The continuous control parameter strategy fusion generates a target intelligent device control strategy parameter set, including: selecting candidate basis vectors whose discrete control parameter setpoints are the same as the execution state from the strategy matrix; extracting the scene contribution weights corresponding to each candidate basis vector and re-normalizing them to generate combination coefficients; using the combination coefficients to perform linear weighted summation on the corresponding continuous control parameters in each candidate basis vector, and combining the execution state to generate the target intelligent device control strategy parameter set; If no other candidate basis vectors with the same discrete control parameter setpoints and execution states are found from the strategy matrix, the continuous control parameter strategy fusion is stopped, and the continuous control parameters in the single scenario control strategy basis vector corresponding to the maximum scenario contribution weight are directly used as the target intelligent device control strategy parameter set.

2. The intelligent interaction method based on computer vision according to claim 1, characterized in that, The process of flattening each channel of the generated deep convolutional feature map in spatial dimension and performing a one-dimensional discrete cosine transform to extract a preset number of low-frequency components to construct a frequency domain channel descriptor includes: The channels of the deep convolutional feature map are flattened into a one-dimensional sequence in row-major order and the one-dimensional discrete cosine transform is performed independently to generate a frequency domain coefficient sequence. According to the frequency from low to high, the first K low-frequency coefficients of the frequency domain coefficient sequence are extracted and concatenated into a vector of length K, which serves as the frequency domain channel descriptor for the corresponding channel; where K is a preset integer greater than 0.

3. The intelligent interaction method based on computer vision according to claim 1, characterized in that, The step of using a one-dimensional convolutional layer to perform cross-channel interactive modeling of the frequency domain channel descriptors and aggregating them in the frequency domain dimension includes: The frequency domain channel descriptors of all channels are concatenated to form a descriptor matrix; After transposing the descriptor matrix, a sliding window convolution is performed on the channel dimension using the one-dimensional convolutional layer, and a flattening operation is applied on the frequency domain dimension to perform dimensional concatenation and aggregation, resulting in a one-dimensional channel feature vector.

4. The intelligent interaction method based on computer vision according to claim 3, characterized in that, The fully connected layer includes a first fully connected layer and a second fully connected layer. The process of sequentially performing dimensional transformation and S-shaped activation mapping through the fully connected layers to generate attention weights for each channel includes: The one-dimensional channel feature vector obtained by aggregation is input into the first fully connected layer to reduce the channel dimension with a preset compression ratio to obtain dimensionality-reduced features. The dimensionality reduction features are input into the second fully connected layer to restore the channel dimension to the same number of channels as the depth convolution feature map; The sigmoid activation mapping is applied to the feature vector after the dimension is restored to generate the attention weights for each channel.

5. The intelligent interaction method based on computer vision according to claim 1, characterized in that, The feature maps are classified to obtain logistic regression scores for each scene category, including: The feature map is input into a global average pooling layer, and average pooling is performed in the spatial dimension to generate a global feature vector. The global feature vector is input into a fully connected classifier, and the logistic regression score of each scene category is obtained by linear transformation through the weight matrix of the fully connected classifier.

6. The intelligent interaction method based on computer vision according to claim 5, characterized in that, The probability distribution vector of the scene category is obtained by applying the Softmax function to the logistic regression score of each scene category for probability normalization.

7. The intelligent interaction method based on computer vision according to claim 1, characterized in that, The scene contribution weight is generated by multiplying the feature contribution vector element-wise with the scene category probability distribution vector and normalizing the result, including: The modulation vector is obtained by performing a Hadamard product operation between the feature contribution vector of each scene and the probability distribution vector of the scene category; The scene contribution weight is generated by summing all elements in the modulation vector and dividing each element of the modulation vector by the sum.

8. A computer vision-based intelligent interactive system, characterized in that, include: A processor and a memory, the memory storing computer program instructions that, when executed by the processor, implement the computer vision-based intelligent interaction method according to any one of claims 1-7.

Images