Smart home system control method, device, server and storage medium

Abstract

The present invention discloses a smart home system control method, device, server and storage medium. The method comprises: calculating the three-dimensional coordinates of each pixel point in an image including a person; based on the position coordinates and sensor values of each environmental sensor of the same type, using Kalman spatial filtering to predict, to obtain sensor prediction values of key points in the image; normalizing the sensor prediction values of the key points to generate an incomplete prediction matrix; inputting the incomplete prediction matrix into a trained super-resolution enhancement neural network model to obtain an expanded complete prediction matrix; fusing the expanded complete prediction matrix of each type with the image to generate a spatial multidimensional vector; performing bilinear pooling on the spatial multidimensional vector and the voice image to obtain N 2 dimensional vector; the N 2 The dimensional vector is input into the trained GCN graph convolutional network model to obtain the control instruction result, and the smart home system is controlled according to the control instruction.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of smart home technology, and more particularly to a smart home system control method, device, server, and storage medium. Background Art

[0002] With the continuous development of technology and the continuous improvement of people's quality of life, people's demand for smart homes is also increasing. Existing smart homes mostly use mobile terminals, voice and other methods to actively send commands, which cannot achieve automatic execution of smart devices.

[0003] With the development of AI technology, the use of neural networks to process various collected information such as environment, voice, images, etc. to realize the automatic execution of smart devices has gradually become the main way to control smart home systems.

[0004] During the implementation of this invention, the inventors discovered the following technical problem: For smart home control, the fusion of multiple types of information can effectively improve the accuracy of predictive control strategies. However, due to the diverse nature of these types of information, current neural networks generally perform feature extraction on each type of information separately through a convolutional layer, followed by a final fully connected layer to generate prediction results. However, this approach ignores the correlation between the various types of information and fails to extract the underlying features, thereby reducing the accuracy of the output results. Summary of the Invention

[0005] Embodiments of the present invention provide a smart home system control method, device, server, and storage medium to solve the technical problem in the prior art that the implicit relationship between various data cannot be extracted, resulting in low accuracy of control results.

[0006] In a first aspect, an embodiment of the present invention provides a smart home system control method, comprising:

[0007] Calculate the three-dimensional coordinates of each pixel in the image including the person in the room;

[0008] Based on the position coordinates and sensor values of each environmental sensor of the same type, a Kalman spatial filter prediction method is used to obtain a sensor prediction value of the key point in the image;

[0009] Normalizing the sensor prediction values of the key points, and generating an incomplete prediction matrix based on the normalized sensor prediction values;

[0010] Inputting the incomplete prediction matrix into the trained super-resolution enhancement neural network model to obtain an expanded complete prediction matrix, wherein the expanded complete prediction matrix has the same size as the image;

[0011] The expanded complete prediction matrix and image of each category are fused to generate a spatial multi-dimensional vector;

[0012] Perform bilinear pooling on the spatial multidimensional vector and the speech image to obtain N 2 dimensional vector;

[0013] The N2-dimensional vector is input into the trained GCN graph convolutional network model to obtain a control instruction result, and the smart home system is controlled according to the control instruction.

[0014] In a second aspect, an embodiment of the present invention further provides a smart home system control device, comprising:

[0015] A three-dimensional coordinate calculation module, used to calculate the three-dimensional coordinates of each pixel point in the image including the person in the room;

[0016] A prediction module, configured to obtain a sensor prediction value corresponding to a key point in the image using a Kalman spatial filter prediction method based on the position coordinates and sensor values of each environmental sensor of the same type;

[0017] an incomplete prediction matrix generating module, configured to normalize the sensor prediction values of the key points and generate an incomplete prediction matrix based on the normalized sensor prediction values;

[0018] An input module is used to input the incomplete prediction matrix into the trained super-resolution enhancement neural network model to obtain an expanded complete prediction matrix, wherein the expanded complete prediction matrix has the same size as the image;

[0019] The fusion module fuses the expanded complete prediction matrix and image of each type to generate a spatial multidimensional vector;

[0020] The pooling module is used to perform bilinear pooling on the spatial multidimensional vector and the speech image to obtain N 2 dimensional vector;

[0021] The control module is used to input the N2-dimensional vector into the trained GCN graph convolutional network model to obtain a control instruction result and control the smart home system according to the control instruction.

[0022] In a third aspect, an embodiment of the present invention further provides a server, including:

[0023] one or more processors;

[0024] a storage device for storing one or more programs;

[0025] When the one or more programs are executed by the one or more processors, the one or more processors implement the smart home system control method provided in the above embodiment.

[0026] In a fourth aspect, an embodiment of the present invention further provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to execute the smart home system control method provided in the above embodiment.

[0027] The smart home system control method, device, server and storage medium provided by the embodiments of the present invention calculate the three-dimensional coordinates of each pixel point in the room of an image including a person; based on the position coordinates and sensor values of each environmental sensor of the same type, use the Kalman spatial filtering prediction method to obtain sensor prediction values of key points in the image; normalize the sensor prediction values of the key points, and generate an incomplete prediction matrix based on the normalized sensor prediction values; input the incomplete prediction matrix into a trained super-resolution enhancement neural network model to obtain an expanded complete prediction matrix, and the expanded complete prediction matrix is the same size as the image; fuse the expanded complete prediction matrix of each type with the image to generate a spatial multidimensional vector; perform bilinear pooling on the spatial multidimensional vector and the speech image to obtain N 2 dimensional vector; the N 2 The dimensional vector is input into the trained GCN graph convolutional network model to obtain the control command result, and the smart home system is controlled according to the control command. By performing multimodal learning on multiple information such as speech, semantics, behavior, video images, etc., and fusing multiple features of various information, more accurate execution judgment results are obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0029] Figure 1 A schematic diagram of a flow chart of a smart home system control method provided in Embodiment 1 of the present invention;

[0030] Figure 2 A schematic diagram of a flow chart of a smart home system control method provided in the second embodiment of the present invention;

[0031] Figure 3 A schematic diagram of a flow chart of a smart home system control method provided in Embodiment 3 of the present invention;

[0032] Figure 4 A schematic diagram of the structure of a smart home system control device provided in Embodiment 4 of the present invention;

[0033] Figure 5 A schematic diagram of the structure of a server provided in Example 5 of the present invention. DETAILED DESCRIPTION

[0034] The present invention will be further described in detail below with reference to the accompanying drawings and examples. It will be understood that the specific embodiments described herein are intended only to illustrate the present invention and are not intended to limit the present invention. It should also be noted that, for ease of description, the accompanying drawings only illustrate portions relevant to the present invention, not all structures.

[0035] Example 1

[0036] Figure 1 This is a flow chart of a smart home system control method provided in Embodiment 1 of the present invention. This embodiment is applicable to controlling a smart home system by fully integrating various features based on data collected by multiple sensors. The method can be executed by a smart home system control device and specifically includes the following steps:

[0037] Step 110: Calculate the three-dimensional coordinates of each pixel point in the image including the person in the room.

[0038] In this embodiment, the smart home control system may include: a voice acquisition device, an image acquisition device, an environmental acquisition device, etc. There may be multiple devices of each type, distributed at multiple locations indoors. The environmental acquisition device may be a variety of sensors, such as a temperature sensor, a humidity sensor, a light sensor, and an air quality sensor.

[0039] Images captured by the image acquisition device must include at least one complete image of a person, primarily capturing the body's form, movements, and expressions. This facilitates the acquisition of complete user motion information. The image acquisition device may include a camera with motion capture and tracking capabilities to ensure that each captured image includes a complete image of the user.

[0040] Calculating the three-dimensional coordinates of each pixel in an image of a person within the room can include: using the position of the image capture device as the viewpoint; using the shooting angle at the image capture location as the viewing angle range; calculating the depth of each pixel based on the position of the fixed object within the room and the depth of field; and calculating the height and width of each pixel within the room based on the viewpoint and viewing angle range. The specific three-dimensional coordinates of the current image within the overall room environment are determined based on the camera position, the current camera angle, and the depth of field. This approach facilitates subsequent fusion with sensor data.

[0041] Step 120 : Based on the position coordinates and sensor values of each environmental sensor of the same type, a Kalman spatial filter prediction method is used to obtain sensor prediction values corresponding to key points in the image.

[0042] Get the properties of each environmental sensor in the indoor environment. The properties can be the type of collected data, such as environmental parameters such as temperature, humidity, and light.

[0043] The spatial coordinates of each environmental sensor are obtained. Environmental sensors are typically located at multiple locations within a room, and some are not within the image. Therefore, the data collected by the sensors must be initially processed to obtain sensor data corresponding to each point within the image.

[0044] Specifically, the sensor data is first classified by type. For each sensor type, the sensor closest to the corresponding center in the image is selected as the most important sensor. Multi-point prediction results are then generated based on the sensor data of the same type collected at multiple points. For temperature, for example, the prediction matrix in the Kalman filter is used to calculate the temperature prediction results between sensors. Unlike traditional Kalman filtering, the time dimension in the Kalman filter is adjusted to the corresponding spatial dimension. Predicted values are then obtained for some points in space.

[0045] Using the above method, the sensor prediction values of the key points of each sensor can be obtained in turn.

[0046] Step 130 : normalize the sensor prediction values of the key points, and generate an incomplete prediction matrix based on the normalized sensor prediction values.

[0047] Since the prediction results obtained through the above steps still do not satisfy a one-to-one correspondence with the image pixels, further processing is required. Optionally, the temperature values can be normalized to form a corresponding matrix. This facilitates further optimization and addition of data based on the characteristics of adjacent matrix elements.

[0048] Step 140: Input the incomplete prediction matrix into the trained super-resolution enhancement neural network model to obtain an expanded complete prediction matrix, where the expanded complete prediction matrix has the same size as the image.

[0049] The Super-Resolution CNN (SRCNN) is the first model to apply deep learning to super-resolution. Its principle is relatively simple. It consists of three neural network layers: Feature Extraction: A low-resolution image is subjected to binomial interpolation to obtain a blurred image, from which image features are extracted. The channel number is 3, the convolution kernel size is f1*f1, and the number of convolution kernels is n1. Nonlinear Mapping: Low-resolution image features are mapped to high resolution with a convolution kernel size of 1*1. Image Reconstruction: Details are restored to obtain a clear high-resolution image with a convolution kernel size of f3*f3. Using an existing super-resolution enhancement neural network model and the principle of close point variation, the temperature matrix is expanded to obtain a temperature matrix that corresponds one-to-one with the image pixels.

[0050] By using the above method, corresponding matrices of various types of data such as humidity and illuminance can also be obtained.

[0051] Step 150: Fuse the expanded complete prediction matrix of each type with the image to generate a spatial multi-dimensional vector.

[0052] The image is converted to grayscale, and an image matrix is constructed for each pixel. The temperature, humidity, and light matrices mentioned above are multiplied with the image matrix to produce a multidimensional tensor. The multidimensionality is n+1, where n is the number of sensor types. Kalman filtering is used to predict the various sensor data, filling in regional data. This data is then further expanded using the existing super-resolution neural network model, utilizing the principle of close proximity of changes in similar points, to ensure a matching relationship with the image pixels and facilitate fusion into a multidimensional tensor.

[0053] Step 160: Perform bilinear pooling on the spatial multidimensional vector and the speech image to obtain N 2 dimensional vector.

[0054] Bilinear pooling is mainly used for feature fusion and then for classification. If feature x and feature y come from two feature extractors, it is called multimodal bilinear pooling (MBP). Bilinear pooling obtains second-order statistical information by taking the outer product, and is intended to use the difference in second-order information for classification when the first-order information is the same. It is convenient for feature extraction from different information sources. In this embodiment, the multidimensional vector and the speech image can be bilinearly pooled to obtain N 2 dimensional vector.

[0055] Step 170: 2 The dimensional vector is input into the trained GCN graph convolutional network model to obtain the control instruction result, and the smart home system is controlled according to the control instruction.

[0056] GCN is suitable for effective feature extraction from graph data. It uses a special method to extract features from graph data, and then uses these features to perform node classification, graph classification, and link prediction on the graph data, and can also obtain graph embedding.

[0057] For example, the above method finally yields an N 2 dimensional vector space, in which the characteristics of the node graph are fully preserved, and multi-dimensional features such as edges and points exist. Using GCN, different control instruction classification results can be obtained. And the above control instructions can be used to effectively control the smart home system.

[0058] This embodiment calculates the three-dimensional coordinates of each pixel point in the room in an image including a person; based on the position coordinates and sensor values of each environmental sensor of the same type, a Kalman spatial filter prediction method is used to obtain sensor prediction values of key points in the image; the sensor prediction values of the key points are normalized, and an incomplete prediction matrix is generated based on the normalized sensor prediction values; the incomplete prediction matrix is input into a trained super-resolution enhancement neural network model to obtain an expanded complete prediction matrix, and the expanded complete prediction matrix is the same size as the image; the expanded complete prediction matrix of each type is fused with the image to generate a spatial multidimensional vector; the spatial multidimensional vector is bilinearly pooled with the speech image to obtain an N2-dimensional vector; the N2-dimensional vector is input into a trained GCN graph convolutional network model to obtain a control instruction result, and the smart home system is controlled according to the control instruction.

[0059] Example 2

[0060] Figure 2 A flowchart of a smart home system control method provided in Example 2 of the present invention. This embodiment is optimized based on the above embodiment. In this embodiment, the method may further include the following steps: determining whether a person has continuously passed through multiple spaces based on images collected at multiple different locations; when continuously passing through multiple spaces, extracting associated features of spatial images passing through multiple areas using a spatial sparse jump method to generate a jump image as the image; accordingly, inputting the incomplete prediction matrix into a trained super-resolution enhancement neural network model to obtain an expanded complete prediction matrix, the expanded complete prediction matrix having the same size as the image. The specific optimization is: generating a sparse measurement matrix as the expanded complete prediction matrix based on the sensor data combination of multiple spaces.

[0061] Accordingly, the smart home system control method provided in this embodiment specifically includes:

[0062] Step 210: Determine whether the person has passed through multiple spaces continuously based on images collected from multiple different positions.

[0063] Step 220 : When continuously passing through multiple spaces, a spatial sparse continuous jump method is used to extract associated features of spatial images passing through multiple areas, and a continuous jump image is generated as an image.

[0064] Considering that when users spend a long time in the same small area, the rate of change of surrounding environmental parameters is low, and the adjustment range and frequency of smart home devices will be less. Therefore, the multidimensional tensor obtained by the above fusion is more suitable for such scenarios.

[0065] On the other hand, if the user enters the area from another environment, the environmental changes will cause greater discomfort to the human body, and the adjustment range and frequency of home equipment will increase. If the multi-dimensional tensor and voice fusion are still used, the output control results will have a large error.

[0066] Therefore, in this embodiment, a spatial-sparse skip connection method is adopted to correct it to obtain a more accurate fusion result.

[0067] Specifically, it can be expressed using the following formula:

[0068] h t (E) =f (E) ([x t ;h t-1 (E) ]),

[0069] in is the hidden state of the previous spatial range. f (E) is a nonlinear function.

[0070] f (E) Not only uses the hidden state of the previous spatial range And also use a state with a jump step size s> 1. This gives the formula:

[0071]

[0072] The jump length s is randomly sampled from [1, 10]. However, random sampling may also affect the accuracy. In the present invention, the determination of the jump length s value can be further optimized.

[0073] Specifically, the S value can be determined based on the captured user image and the area before the area. That is, the interval of continuous jumps is the number of separated areas. If the user passes through multiple areas continuously, the farthest area can be selected to determine the S value.

[0074] Step 230: Calculate the three-dimensional coordinates of each pixel point in the image including the person in the room.

[0075] Step 240 : Based on the position coordinates and sensor values of each environmental sensor of the same type, a Kalman spatial filter prediction method is used to obtain sensor prediction values corresponding to key points in the image.

[0076] Step 250: Generate a sparse measurement matrix as an expanded complete prediction matrix based on the sensor data combination of multiple spaces, where the expanded complete prediction matrix has the same size as the image.

[0077] In this case, the super-resolution model can be used to expand the matrix for the jump intervals. Instead, a larger space is used to generate the corresponding sensor parameter matrix using a sparse representation. This approach takes into account that when users continuously traverse the area, they spend less time in smaller areas, so these small areas will not affect the user's sensory perception. Therefore, enlarging the area not only does not affect accuracy, but also significantly reduces the amount of computation required.

[0078] Step 260: Fuse the expanded complete prediction matrix of each type with the image to generate a spatial multi-dimensional vector.

[0079] Step 270: Perform bilinear pooling on the spatial multidimensional vector and the speech image to obtain N 2 dimensional vector.

[0080] Step 280: 2 The dimensional vector is input into the trained GCN graph convolutional network model to obtain the control instruction result, and the smart home system is controlled according to the control instruction.

[0081] This embodiment adds the following steps: determining whether a person has continuously passed through multiple spaces based on images collected at multiple different locations; when a person has continuously passed through multiple spaces, using a spatial sparse jump method to extract the associated features of the spatial images passing through multiple regions, generating a jump image as an image; accordingly, inputting the incomplete prediction matrix into a trained super-resolution enhancement neural network model to obtain an expanded complete prediction matrix, which has the same size as the image. The specific optimization is: generating a sparse measurement matrix as the expanded complete prediction matrix based on the sensor data combination of multiple spaces. Using the spatial-sparse jump method, the sensor data of multiple regions is fused to facilitate the acquisition of hidden states, fully reflect the sensory experience characteristics of the moving human body, improve the accuracy of the prediction execution results for the moving human body, and thus enhance the user experience. The jump length can also be optimized and adjusted according to the movement speed, effectively reducing the amount of calculation without reducing the accuracy.

[0082] Example 3

[0083] Figure 3 This is a flow chart of a smart home system control method provided by Embodiment 3 of the present invention. This embodiment is optimized based on the above embodiment. In this embodiment, the method may further include the following step: converting speech into speech image using Mel-frequency cepstral coefficients.

[0084] Accordingly, the smart home system control method provided in this embodiment specifically includes:

[0085] Step 310: Calculate the three-dimensional coordinates of each pixel point in the image including the person in the room.

[0086] Step 320: Based on the position coordinates and sensor values of each environmental sensor of the same type, a Kalman spatial filter prediction method is used to obtain sensor prediction values corresponding to key points in the image.

[0087] Step 330 : normalize the sensor prediction values of the key points, and generate an incomplete prediction matrix based on the normalized sensor prediction values.

[0088] Step 340: Input the incomplete prediction matrix into the trained super-resolution enhancement neural network model to obtain an expanded complete prediction matrix, where the expanded complete prediction matrix has the same size as the image.

[0089] Step 350: Fuse the expanded complete prediction matrix of each type with the image to generate a spatial multi-dimensional vector.

[0090] Step 360: Convert the speech into a speech image using the Mel-frequency cepstral coefficients.

[0091] MFCCs (Mel Frequency Cepstral Coefficents) are a feature widely used in automatic speech and speaker recognition. They focus on specific frequency components. Mel Frequency Cepstrum Coefficients (MFCCs) take into account human auditory characteristics, first mapping the linear spectrum to a Mel nonlinear spectrum based on auditory perception, and then converting it to the cepstrum.

[0092] Exemplarily, the method further includes: applying first-order differences to the Mel-frequency cepstral coefficients to incorporate speech features from the previous moment; and using the Mel-frequency cepstral coefficients after first-order differences as the corresponding speech image. This method converts speech features into corresponding images, facilitating feature fusion extraction and fully exploring the correlation features between speech and image. It also fully utilizes the sequential nature of speech to further improve the accuracy of control command classification.

[0093] Step 370: Perform bilinear pooling on the spatial multidimensional vector and the speech image to obtain N 2 dimensional vector.

[0094] Step 380: Input the N2-dimensional vector into the trained GCN graph convolutional network model to obtain a control instruction result, and control the smart home system according to the control instruction.

[0095] This embodiment adds the following step: converting speech into a speech image using Mel-frequency cepstrum coefficients. This allows speech information to be converted into image information, facilitating multi-space fusion with other data, thereby extracting the correlation characteristics between speech and image. Furthermore, using the first-order difference Mel-frequency cepstrum, implicit features between the beginning and end of speech can be extracted, further improving the accuracy of control command classification.

[0096] Example 4

[0097] Figure 4 This is a schematic diagram of the structure of the smart home system control device provided by the fourth embodiment of the present invention, as shown in FIG. Figure 4 As shown, the device includes:

[0098] A three-dimensional coordinate calculation module 410 is used to calculate the three-dimensional coordinates of each pixel point in the image including the person in the room;

[0099] A prediction module 420 is configured to obtain a sensor prediction value corresponding to a key point in the image using a Kalman spatial filter prediction method based on the position coordinates and sensor values of each environmental sensor of the same type;

[0100] an incomplete prediction matrix generating module 430, configured to normalize the sensor prediction values of the key points and generate an incomplete prediction matrix based on the normalized sensor prediction values;

[0101] An input module 440 is configured to input the incomplete prediction matrix into a trained super-resolution enhancement neural network model to obtain an expanded complete prediction matrix, wherein the expanded complete prediction matrix has the same size as the image;

[0102] A fusion module 450 is used to fuse the expanded complete prediction matrix of each type with the image to generate a spatial multi-dimensional vector;

[0103] The pooling module 460 is used to perform bilinear pooling on the spatial multidimensional vector and the speech image to obtain N 2 dimensional vector;

[0104] The control module 470 is used to 2 The dimensional vector is input into the trained GCN graph convolutional network model to obtain the control instruction result, and the smart home system is controlled according to the control instruction.

[0105] The smart home system control device provided by this embodiment calculates the three-dimensional coordinates of each pixel point in the room in an image including a person; based on the position coordinates and sensor values of each environmental sensor of the same type, uses the Kalman spatial filtering prediction method to obtain sensor prediction values of key points in the image; normalizes the sensor prediction values of the key points, and generates an incomplete prediction matrix based on the normalized sensor prediction values; inputs the incomplete prediction matrix into a trained super-resolution enhancement neural network model to obtain an expanded complete prediction matrix, and the expanded complete prediction matrix has the same size as the image; fuses the expanded complete prediction matrix of each type with the image to generate a spatial multidimensional vector; performs bilinear pooling on the spatial multidimensional vector and the voice image to obtain N 2 dimensional vector; the N 2 The dimensional vector is input into the trained GCN graph convolutional network model to obtain the control command result, and the smart home system is controlled according to the control command. By performing multimodal learning on multiple information such as speech, semantics, behavior, video images, etc., and fusing multiple features of various information, more accurate execution judgment results are obtained.

[0106] Based on the above embodiments, the device further includes:

[0107] A judgment module, used to judge whether a person has passed through multiple spaces continuously based on images collected at multiple different positions;

[0108] an extraction module for extracting associated features of spatial images passing through multiple areas by using a spatial sparse continuous jump method when continuously passing through multiple spaces, and generating a continuous jump image as an image;

[0109] Accordingly, the incomplete prediction matrix generation module includes:

[0110] The combination unit is used to generate a sparse measurement matrix as an expanded complete prediction matrix according to the sensor data of multiple spaces.

[0111] Based on the above embodiments, the extraction module is used to:

[0112] This is achieved using the following method:

[0113] h t (E) =f (E) ([x t ;h t-1 (E) ]),

[0114] in is the hidden state of the previous spatial range. f (E) is a nonlinear function;

[0115] f (E) Not only uses the hidden state of the previous spatial range And also use a state with a jump step length s>1, and then get the formula:

[0116]

[0117] Based on the above embodiments, the device further includes:

[0118] The jump value determination module is used to determine the S value according to the captured user image and the area before the area, that is, the continuous jump interval is the number of separated areas.

[0119] Based on the above embodiments, the device further includes:

[0120] The speech image conversion module is used to convert speech into speech image through Mel-frequency cepstral coefficients.

[0121] Based on the above embodiments, the device further includes:

[0122] The difference module is used to perform first-order difference on the Mel-frequency cepstral coefficients in order to incorporate the speech features of the previous moment;

[0123] As a module, it is used to use the Mel frequency cepstral coefficients after first-order difference processing as the corresponding speech image.

[0124] Based on the above embodiments, the calculation module includes:

[0125] a viewpoint unit, configured to use the position of the image acquisition device as a viewpoint;

[0126] A viewing angle range unit, configured to use a shooting angle of the image acquisition position as a viewing angle range;

[0127] a depth calculation unit, configured to calculate the depth of each pixel point based on the position of the fixed object in the indoor space and the depth of field;

[0128] The height and width calculation unit is used to calculate the height and width of each pixel in space based on the viewpoint and viewing angle range.

[0129] The smart home system control device provided in the embodiment of the present invention can execute the smart home system control method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0130] Example 5

[0131] Figure 5 A schematic diagram of the structure of a server provided in Example 5 of the present invention. Figure 5 A block diagram of an exemplary server 12 suitable for use in implementing embodiments of the present invention is shown. Figure 5 The server 12 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0132] like Figure 5 As shown, server 12 is implemented as a general-purpose computing device. Components of server 12 may include, but are not limited to, one or more processors or processing units 16, system memory 28, and a bus 18 that connects various system components (including system memory 28 and processing units 16).

[0133] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures. Examples of these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0134] The server 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the server 12, including volatile and non-volatile media, removable and non-removable media.

[0135] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache 32. Server 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be configured to read and write non-removable, non-volatile magnetic media ( Figure 4 Not shown, often called a "hard drive"). Although Figure 4 Not shown, a magnetic disk drive for reading and writing to a removable non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. System memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of various embodiments of the present invention.

[0136] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may include an implementation of a network environment. Program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0137] The server 12 may also communicate with one or more external devices 14 (e.g., a keyboard, pointing device, display 24, etc.), one or more devices that enable a user to interact with the server 12, and / or any device that enables the server 12 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may occur via an input / output (I / O) interface 22. Furthermore, the server 12 may communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with the other modules of the server 12 via a bus 18. It should be understood that, although not shown, other hardware and / or software modules may be used in conjunction with the server 12, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0138] The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, such as implementing the smart home system control method provided by the embodiment of the present invention.

[0139] Example 6

[0140] Embodiment 6 of the present invention further provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to execute any of the smart home system control methods provided in the above embodiments.

[0141] The computer storage medium of the embodiment of the present invention may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device.

[0142] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0143] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0144] Computer program code for performing the operations of the present invention may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0145] Note that the above are only preferred embodiments of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments, and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments and may include many other equivalent embodiments without departing from the concept of the present invention. The scope of the present invention is determined by the scope of the appended claims.

Claims

1. A smart home system control method, characterized in that: include: Calculate the three-dimensional coordinates of each pixel in the image including the person in the room; Based on the position coordinates and sensor values of each environmental sensor of the same type, a Kalman spatial filter prediction method is used to obtain a sensor prediction value of the key point in the image; Normalizing the sensor prediction values of the key points, and generating an incomplete prediction matrix based on the normalized sensor prediction values; Inputting the incomplete prediction matrix into the trained super-resolution enhancement neural network model to obtain an expanded complete prediction matrix, wherein the expanded complete prediction matrix has the same size as the image; The expanded complete prediction matrix and image of each category are fused to generate a spatial multi-dimensional vector; Perform bilinear pooling on the spatial multidimensional vector and the speech image to obtain N 2 dimensional vector; The N 2 The dimensional vector is input into the trained GCN graph convolutional network model to obtain the control instruction result, and the smart home system is controlled according to the control instruction.

2. The method according to claim 1, characterized in that The method further comprises: Determine whether a person has passed through multiple spaces continuously based on images collected at multiple different locations; When passing through multiple spaces continuously, a spatial sparse jump method is used to extract the associated features of the spatial images passing through multiple areas, and a jump image is generated as an image; Accordingly, the incomplete prediction matrix is input into the trained super-resolution enhancement neural network model to obtain an expanded complete prediction matrix, where the expanded complete prediction matrix has the same size as the image, including: A sparse measurement matrix is generated according to the combination of sensor data in multiple spaces as an augmented complete prediction matrix.

3. The method according to claim 2, characterized in that The method of extracting the associated features of the spatial image passing through multiple regions by using the spatial sparse jump method and generating the jump image as the image is implemented by the following method: h t (E) =f (E) ([x t ;h t-1 (E) ]), in is the hidden state of the previous spatial range, f (E) is a nonlinear function; f (E) Not only uses the hidden state of the previous spatial range And also use a state with a jump step length s>1, and then get the formula:

4. The method according to claim 3, characterized in that The method further comprises: The S value is determined based on the captured user image and the area before the area, that is, the interval of continuous jumps is the number of separated areas.

5. The method according to claim 1, wherein The method further comprises: Convert speech into speech image through Mel-frequency cepstral coefficients.

6. The method according to claim 5, characterized in that The method further comprises: The first-order difference method is used for the Mel frequency cepstral coefficients to incorporate the speech characteristics of the previous moment; The Mel-frequency cepstral coefficients after first-order difference processing are used as the corresponding speech image.

7. The method according to claim 1, characterized in that The calculation includes the three-dimensional coordinates of each pixel point in the image of the person in the room: Taking the position of the image acquisition device as a viewpoint; The shooting angle of the image acquisition position is used as the viewing angle range; Calculate the depth of each pixel based on the position of the fixed object in the indoor space and the depth of field; Calculate the height and width of each pixel in space based on the viewpoint and viewing angle range.

8. A smart home system control device, characterized in that: include: A three-dimensional coordinate calculation module, used to calculate the three-dimensional coordinates of each pixel point in the image including the person in the room; A prediction module, configured to obtain a sensor prediction value corresponding to a key point in the image using a Kalman spatial filter prediction method based on the position coordinates and sensor values of each environmental sensor of the same type; an incomplete prediction matrix generating module, configured to normalize the sensor prediction values of the key points and generate an incomplete prediction matrix based on the normalized sensor prediction values; An input module is used to input the incomplete prediction matrix into the trained super-resolution enhancement neural network model to obtain an expanded complete prediction matrix, wherein the expanded complete prediction matrix has the same size as the image; A fusion module is used to fuse the expanded complete prediction matrix and image of each type to generate a spatial multidimensional vector; The pooling module is used to perform bilinear pooling on the spatial multidimensional vector and the speech image to obtain N 2 dimensional vector; The control module is used to input the N2-dimensional vector into the trained GCN graph convolutional network model to obtain a control instruction result and control the smart home system according to the control instruction.

9. A server, characterized in that: The server includes: one or more processors; a storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the smart home system control method as described in any one of claims 1 to 7.

10. A storage medium comprising computer-executable instructions, wherein the computer-executable instructions, when executed by a computer processor, are used to perform the smart home system control method according to any one of claims 1 to 7.

Images