Electronic equipment adjusting method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides an electronic equipment adjusting method and device, electronic equipment and a storage medium, and is applied to the technical field of smart home. By applying the method provided by the embodiment of the invention, the shoulder key points and the cross key points corresponding to the target person can be determined by performing key point detection on the collected first to-be-processed image, and the shoulder cross image distance is calculated according to the coordinates of the shoulder key points and the cross key points in the image; and the shoulder-crossing image distance is converted into the first distance representing the actual distance between the target person and the electronic equipment, so that the target wind speed can be determined according to the first distance, and the electronic equipment can adjust the wind speed according to the distance between the user and the electronic equipment. Compared with an operation mode in the related technology, the equipment adjusting method provided by the embodiment of the invention has the advantages that a user does not need to carry out key operation, and the operation is more convenient.

Description

Technical Field

[0001] The present application relates to the field of smart home technology, and in particular to an electronic device adjustment method, device, electronic device and storage medium. Background Art

[0002] As smart homes become increasingly popular, people have a growing demand for smart and convenient home appliances. In related technologies, people usually control devices by buttons, knobs, remote controls, etc. However, such control methods usually have certain limitations. For example, to adjust the device by buttons and knobs, the user needs to go to the device to operate; to adjust the device by remote control, the user needs to hold the remote control and aim it at the device to adjust it. This method is obviously not smart enough, so how to enable users to adjust the device more conveniently is an urgent problem to be solved. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide an electronic device adjustment method, device, electronic device and storage medium to achieve more convenient device adjustment. The specific technical solution is as follows:

[0004] In a first aspect of an embodiment of the present application, a method for adjusting an electronic device is provided, wherein the electronic device includes a fan and a camera, and the method includes:

[0005] Acquire a first image to be processed captured by the camera; wherein the first image to be processed includes a target person;

[0006] Performing human body key point detection on the first image to be processed to determine the shoulder key points and crotch key points of the target person in the first image to be processed;

[0007] Calculating a shoulder-span image distance according to image coordinates of a shoulder key point and a crotch key point in the first image to be processed, and converting the shoulder-span image distance into a first distance, wherein the first distance represents a distance between the target person and the electronic device;

[0008] A preset wind speed correspondence relationship is queried to determine a target wind speed corresponding to the first distance, and the fan is adjusted to the target wind speed, wherein the wind speed correspondence relationship is a preset correspondence relationship between wind speed and distance.

[0009] In a possible implementation manner, the step of calculating the shoulder-span image distance according to the image coordinates of the shoulder key points and the crotch key points in the first image to be processed, and converting the shoulder-span image distance into the first distance comprises:

[0010] The coordinates of the shoulder key points and / or crotch key points are calculated by the following formula:

[0011] ;

[0012] ;

[0013] Among them, the k x represents the horizontal coordinate of the shoulder key point or the crotch key point, the k y The ordinate of the shoulder key point or the crotch key point is represented by the horizontal and vertical coordinates of the offset coordinates of the key point predicted by the first neural network model relative to the anchor point. kx and t ky The horizontal and vertical coordinates of the anchor point coordinates are c x and c y ; The strides represent the downsampling scaling ratio;

[0014] The coordinates of the center points of the two shoulder key points and the center points of the two span key points are calculated using the following formula:

[0015] ;

[0016] Among them, the k csx represents the horizontal coordinate of the center point of the two shoulder key points, the k lsx represents the horizontal coordinate of the left shoulder key point of the target person, and the k rsx The horizontal coordinate of the key point of the right shoulder of the target person;

[0017] ;

[0018] Among them, the k csy represents the ordinate of the center point of the two shoulder key points, the k lsy represents the ordinate of the left shoulder key point of the target person, and the k rsy The vertical coordinate of the key point of the right shoulder of the target person;

[0019] ;

[0020] Among them, the k chx represents the horizontal coordinate of the center point of the two cross-part key points, the k lhx represents the horizontal coordinate of the left span key point of the target person, and the k rhx The horizontal coordinate of the right span key point of the target person;

[0021] ;

[0022] Among them, the k chy represents the ordinate of the center point of the two cross-part key points, the k lhyrepresents the ordinate of the left span key point of the target person, and the k rhy The ordinate of the right-span key point corresponding to the target;

[0023] According to the coordinates of the center points of the two shoulder key points and the coordinates of the center points of the two span key points, the shoulder-span image distance is calculated by the following formula;

[0024] ;

[0025] Among them, the d pix represents the shoulder-span image distance;

[0026] The shoulder-span image distance is converted into the first distance by the following formula;

[0027] ;

[0028] Wherein, D represents the first distance, f represents the focal length of the camera, and d body Indicates the preset average shoulder span distance.

[0029] In a possible implementation, the performing of human body key point detection on the first image to be processed to determine the shoulder key points and crotch key points of the target person in the first image to be processed includes:

[0030] Inputting the first image to be processed into a pre-trained first neural network model to obtain shoulder key points and crotch key points of the target person in the first image to be processed;

[0031] Wherein, the first neural network model further outputs a gesture frame of the target person, and the method further includes:

[0032] According to the gesture frame, the first image to be processed is segmented to obtain a gesture image;

[0033] Inputting the gesture image into a pre-trained second neural network model to obtain a gesture classification result corresponding to the gesture image, wherein the gesture classification result indicates a business gesture category corresponding to the gesture in the gesture image;

[0034] According to the gesture classification result, a device control instruction is determined to enable the fan to perform an adjustment operation corresponding to the device control instruction.

[0035] In a possible implementation, before inputting the gesture image into a pre-trained second neural network model to obtain a gesture classification result corresponding to the gesture image, the method further includes:

[0036] Inputting the gesture image into a pre-trained third neural network model to obtain a clarity score and a blur score of the gesture image; wherein the third neural network model is used to perform clarity detection on the gesture image;

[0037] When the blur score is greater than or equal to the clarity score, discarding the gesture image;

[0038] In the case where the clarity score is greater than the fuzziness score, the step of inputting the gesture image into a pre-trained second neural network model is performed to obtain a gesture classification result corresponding to the gesture image.

[0039] In a possible implementation manner, after acquiring the first image to be processed collected by the fan, the method further includes:

[0040] Acquire a second image to be processed captured by the fan, wherein the second image to be processed is an image captured after the first image to be processed;

[0041] Inputting the first image to be processed and the second image to be processed into a first neural network model respectively to obtain a first target frame and a second target frame, wherein the first target frame represents the position of the target person in the first image to be processed, and the second target frame represents the position of the target person in the second image to be processed;

[0042] Calculating a third target frame of the target person according to the first target frame, wherein the third target frame represents a predicted position of the target person in the second image to be processed;

[0043] Matching the second target frame with the third target frame in positional relationship to obtain a matching result;

[0044] When the matching result indicates that the target person in the first image to be processed and the target person in the second image to be processed are the same target person, calculating the difference between the first target frame and the second target frame to obtain position transformation information;

[0045] According to the position change information, a device adjustment instruction is determined so that the fan is adjusted according to the device adjustment instruction.

[0046] In a possible implementation, calculating a third target frame of the target person according to the first target frame includes:

[0047] Calculating an initial state vector of the target person in the first image to be processed according to the first target frame;

[0048] Calculating, based on the initial state vector, an error between the first predicted state vector and the first predicted state vector;

[0049] Calculating the error between the measurement value of the target person in the second to-be-detected image and the measurement value according to the measurement matrix of the measuring device;

[0050] Calculating a Kalman gain by combining an error in the measurement value with an error in the first predicted state vector;

[0051] Correcting the first predicted state vector by using the Kalman gain to obtain a first corrected state vector;

[0052] The third target frame is calculated according to the first corrected state vector.

[0053] In a possible implementation, performing positional relationship matching between the second target frame and the third target frame to obtain a matching result includes:

[0054] Calculating an intersection-over-union ratio of the second target frame and the third target frame;

[0055] When the intersection-over-union ratio is greater than or equal to a preset threshold, it is determined that the matching result indicates that the target person in the first image to be processed and the target person in the second image to be processed are the same target person.

[0056] In a possible implementation, inputting the first image to be processed into a pre-trained first neural network model to obtain the shoulder key points and crotch key points of the target person in the first image to be processed includes:

[0057] Inputting the first image to be processed into the first neural network model, wherein the first network structure in the first neural network model performs feature extraction and downsampling processing on the first image to be processed to obtain first image features;

[0058] Inputting the first image feature into a second network structure for convolution processing, feature enhancement processing and downsampling processing to obtain a second image feature; wherein the second network structure is the same as the first network structure;

[0059] Inputting the second image feature into the third network structure for feature enhancement processing to obtain a third image feature;

[0060] Inputting the third image feature into a fourth network structure for convolution processing, feature enhancement processing and downsampling processing to obtain a fourth image feature; wherein the fourth network structure is the same as the second network structure;

[0061] Inputting the fourth image feature into a fifth network structure for feature enhancement processing to obtain a fifth image feature; wherein the fifth network structure is the same as the third network structure;

[0062] Inputting the fifth image feature into a sixth network structure for convolution processing, feature enhancement processing and downsampling processing to obtain a sixth image feature, wherein the sixth network structure is the same as the fourth network structure;

[0063] Inputting the sixth image feature into a seventh network structure for feature enhancement processing to obtain a seventh image feature, wherein the seventh network structure is the same as the fifth network structure;

[0064] Inputting the seventh image feature into an eighth network structure for convolution processing, feature enhancement processing and downsampling processing to obtain an eighth image feature; wherein the eighth network structure is the same as the sixth network structure;

[0065] Inputting the eighth image feature into a ninth network structure to perform a multi-scale pooling operation, fusing image features of different scales, and obtaining a ninth image feature;

[0066] Inputting the ninth image feature into a tenth network structure for convolution processing and feature enhancement processing to obtain a tenth image feature, wherein the tenth image feature, the seventh image feature, and the fifth image feature have different sizes;

[0067] Performing convolution processing and feature enhancement processing on the tenth image feature through an eleventh network structure to obtain an eleventh image feature, wherein the eleventh network structure is the same as the network structure of the eighth network structure;

[0068] Performing upsampling processing on the eleventh image feature by a first upsampling module to obtain a twelfth image feature, wherein the twelfth image feature has the same size as the seventh image feature;

[0069] The twelfth image feature and the seventh image feature are fused through a first feature fusion layer to obtain a thirteenth image feature;

[0070] Inputting the thirteenth image feature into a twelfth network structure for convolution processing and feature enhancement processing to obtain a fourteenth image feature, wherein the twelfth network structure has the same network structure as the tenth network structure;

[0071] Inputting the fourteenth image feature into the thirteenth network structure for convolution processing and feature enhancement processing to obtain a fifteenth image feature, wherein the thirteenth network structure is the same as the eleventh network structure;

[0072] performing upsampling processing on the fifteenth image feature through a second upsampling module to obtain a sixteenth image feature, wherein the sixteenth image feature has the same size as the fifth image feature;

[0073] The fifth image feature and the sixteenth image feature are subjected to feature fusion by a second feature fusion layer to obtain a seventeenth image feature;

[0074] Inputting the seventeenth image feature into a fourteenth network structure for convolution processing and feature enhancement processing to obtain an eighteenth image feature; wherein the fourteenth network structure has the same network structure as the twelfth network structure;

[0075] Inputting the eighteenth image feature into a fifteenth network structure for convolution processing, feature enhancement processing and downsampling processing to obtain a nineteenth image feature, wherein the fifteenth network structure is the same as the thirteenth network structure;

[0076] The nineteenth image feature is fused with the fifteenth image feature through a third feature fusion layer to obtain a twentieth image feature;

[0077] Inputting the 20th image feature into a 16th network structure for convolution processing and feature enhancement processing to obtain a 21st image feature, wherein the 16th network structure is the same as the 14th network structure;

[0078] Inputting the twenty-first image feature into a seventeenth network structure for convolution processing, feature enhancement processing, and downsampling processing to obtain a twenty-second image feature, wherein the seventeenth network structure is the same as the fifteenth network structure;

[0079] The twenty-second image feature and the eleventh image feature are fused through a fourth feature fusion layer to obtain a twenty-third image feature;

[0080] Inputting the twenty-third image feature into an eighteenth network structure for convolution processing and feature enhancement processing to obtain a twenty-fourth image feature, wherein the eighteenth network structure is the same as the sixteenth network structure;

[0081] The twenty-fourth image feature, the twenty-first image feature and the eighteenth image feature are respectively input into the network head modules corresponding to their respective sizes for classification prediction, so as to obtain the shoulder key points and crotch key points of the target person.

[0082] According to a second aspect of an embodiment of the present application, there is provided an electronic device adjustment device, wherein the electronic device includes a fan and a camera, and the device includes:

[0083] A first image to be processed acquisition module, used to acquire a first image to be processed captured by the camera; wherein the first image to be processed includes a target person;

[0084] A key point detection module, used to perform human key point detection on the first image to be processed, and determine the shoulder key points and crotch key points of the target person in the first image to be processed;

[0085] A first distance calculation module, configured to calculate a shoulder-span image distance according to image coordinates of shoulder key points and crotch key points in the first image to be processed, and convert the shoulder-span image distance into a first distance, wherein the first distance represents a distance between the target person and the electronic device;

[0086] The target wind speed determination module is used to query a preset wind speed correspondence, determine a target wind speed corresponding to the first distance, and adjust the fan to the target wind speed, wherein the wind speed correspondence is a preset correspondence between wind speed and distance.

[0087] In a possible implementation manner, the first distance calculation module includes:

[0088] The key point coordinate calculation submodule is specifically used to calculate the coordinates of the shoulder key point and / or the crotch key point by the following formula:

[0089] ;

[0090] ;

[0091] Among them, the k x represents the horizontal coordinate of the shoulder key point or the crotch key point, the k y The ordinate of the shoulder key point or the crotch key point is represented by the horizontal and vertical coordinates of the offset coordinates of the key point predicted by the first neural network model relative to the anchor point. kx and t ky The horizontal and vertical coordinates of the anchor point coordinates are c x and c y ; The strides represent the downsampling scaling ratio;

[0092] The center point coordinate calculation submodule is specifically used to calculate the coordinates of the center points of the two shoulder key points and the coordinates of the center points of the two span key points through the following formula:

[0093] ;

[0094] Among them, the k csx represents the horizontal coordinate of the center point of the two shoulder key points, the k lsxrepresents the horizontal coordinate of the left shoulder key point of the target person, and the k rsx The horizontal coordinate of the key point of the right shoulder of the target person;

[0095] ;

[0096] Among them, the k csy represents the ordinate of the center point of the two shoulder key points, the k lsy represents the ordinate of the left shoulder key point of the target person, and the k rsy The vertical coordinate of the key point of the right shoulder of the target person;

[0097] ;

[0098] Among them, the k chx represents the horizontal coordinate of the center point of the two cross-part key points, the k lhx represents the horizontal coordinate of the left span key point of the target person, and the k rhx The horizontal coordinate of the right span key point of the target person;

[0099] ;

[0100] Among them, the k chy represents the ordinate of the center point of the two cross-part key points, the k lhy represents the ordinate of the left span key point of the target person, and the k rhy The ordinate of the right-span key point corresponding to the target;

[0101] The shoulder-span image distance calculation submodule is specifically used to calculate the shoulder-span image distance according to the coordinates of the center points of the two shoulder key points and the coordinates of the center points of the two span key points by the following formula;

[0102] ;

[0103] Among them, the d pix represents the shoulder-span image distance;

[0104] A first distance calculation submodule is specifically used to convert the shoulder span image distance into the first distance by using the following formula;

[0105] ;

[0106] Wherein, D represents the first distance, f represents the focal length of the camera, and d body Indicates the preset average shoulder span distance.

[0107] In a possible implementation, the key point detection module includes:

[0108] A first neural network model prediction submodule, specifically used to input the first image to be processed into a pre-trained first neural network model to obtain the shoulder key points and crotch key points of the target person in the first image to be processed;

[0109] Wherein, the first neural network model further outputs a gesture frame of the target person, and the device further comprises:

[0110] An image cutting module, used for cutting the first image to be processed according to the gesture frame to obtain a gesture image;

[0111] A gesture detection module, used for inputting the gesture image into a pre-trained second neural network model to obtain a gesture classification result corresponding to the gesture image, wherein the gesture classification result indicates a business gesture category corresponding to the gesture in the gesture image;

[0112] The device control instruction determination module is used to determine the device control instruction according to the gesture classification result so as to make the fan perform the adjustment operation corresponding to the device control instruction.

[0113] In a possible implementation, the device further includes:

[0114] A clarity detection module, used for inputting the gesture image into a pre-trained third neural network model to obtain a clarity score and a blur score of the gesture image; wherein the third neural network model is used for performing clarity detection on the gesture image;

[0115] A fuzzy image processing module, configured to discard the gesture image if the fuzziness score is greater than or equal to the clarity score;

[0116] The clear image processing module is used to execute the steps of: inputting the gesture image into a pre-trained second neural network model to obtain a gesture classification result corresponding to the gesture image when the clarity score is greater than the blur score.

[0117] In a possible implementation, the device further includes:

[0118] A second image to be processed acquisition module, used to acquire a second image to be processed acquired by the fan, wherein the second image to be processed is an image acquired after the first image to be processed;

[0119] a target frame acquisition module, used to input the first image to be processed and the second image to be processed into a first neural network model respectively, to obtain a first target frame and a second target frame, wherein the first target frame represents the position of the target person in the first image to be processed, and the second target frame represents the position of the target person in the second image to be processed;

[0120] A third target frame prediction module, used to calculate a third target frame of the target person according to the first target frame, wherein the third target frame represents a predicted position of the target person in the second image to be processed;

[0121] A position relationship matching module, used for matching the position relationship between the second target frame and the third target frame to obtain a matching result;

[0122] a position transformation information calculation module, configured to calculate a difference between the first target frame and the second target frame to obtain position transformation information when the matching result indicates that the target person in the first image to be processed and the target person in the second image to be processed are the same target person;

[0123] The device adjustment instruction determination module is used to determine the device adjustment instruction according to the position change information so that the fan is adjusted according to the device adjustment instruction.

[0124] In a possible implementation, the third target frame prediction module includes:

[0125] An initial state vector calculation submodule, specifically used to calculate the initial state vector of the target person in the first image to be processed according to the first target frame;

[0126] A prediction vector calculation submodule, specifically used to calculate the error between the first prediction state vector and the first prediction state vector according to the initial state vector;

[0127] A measurement value calculation submodule, specifically used to calculate the measurement value of the target person in the second to-be-detected image and the error between the measurement value according to the measurement matrix of the measuring device;

[0128] A Kalman gain calculation submodule, specifically used to calculate the Kalman gain by combining the error of the measurement value with the error of the first predicted state vector;

[0129] A prediction vector correction submodule, specifically used to correct the first prediction state vector by using the Kalman gain to obtain a first corrected state vector;

[0130] The third target frame calculation submodule is specifically used to calculate the third target frame according to the first corrected state vector.

[0131] In a possible implementation, the position relationship matching module includes:

[0132] An intersection-and-union ratio calculation submodule, specifically used to calculate the intersection-and-union ratio of the second target frame and the third target frame;

[0133] The intersection-and-union ratio judgment submodule is specifically used to determine that the matching result indicates that the target person in the first image to be processed and the target person in the second image to be processed are the same target person when the intersection-and-union ratio is greater than or equal to a preset threshold.

[0134] In a possible implementation, the key point detection module includes:

[0135] An image processing submodule, specifically configured to input the first image to be processed into the first neural network model, wherein the first network structure in the first neural network model performs feature extraction and downsampling processing on the first image to be processed to obtain first image features;

[0136] Inputting the first image feature into a second network structure for convolution processing, feature enhancement processing and downsampling processing to obtain a second image feature; wherein the second network structure is the same as the first network structure;

[0137] Inputting the second image feature into the third network structure for feature enhancement processing to obtain a third image feature;

[0138] Inputting the third image feature into a fourth network structure for convolution processing, feature enhancement processing and downsampling processing to obtain a fourth image feature; wherein the fourth network structure is the same as the second network structure;

[0139] Inputting the fourth image feature into a fifth network structure for feature enhancement processing to obtain a fifth image feature; wherein the fifth network structure is the same as the third network structure;

[0140] Inputting the fifth image feature into a sixth network structure for convolution processing, feature enhancement processing and downsampling processing to obtain a sixth image feature, wherein the sixth network structure is the same as the fourth network structure;

[0141] Inputting the sixth image feature into a seventh network structure for feature enhancement processing to obtain a seventh image feature, wherein the seventh network structure is the same as the fifth network structure;

[0142] Inputting the seventh image feature into an eighth network structure for convolution processing, feature enhancement processing and downsampling processing to obtain an eighth image feature; wherein the eighth network structure is the same as the sixth network structure;

[0143] Inputting the eighth image feature into a ninth network structure to perform a multi-scale pooling operation, fusing image features of different scales, and obtaining a ninth image feature;

[0144] Inputting the ninth image feature into a tenth network structure for convolution processing and feature enhancement processing to obtain a tenth image feature, wherein the tenth image feature, the seventh image feature, and the fifth image feature have different sizes;

[0145] Performing convolution processing and feature enhancement processing on the tenth image feature through an eleventh network structure to obtain an eleventh image feature, wherein the eleventh network structure is the same as the network structure of the eighth network structure;

[0146] Performing upsampling processing on the eleventh image feature by a first upsampling module to obtain a twelfth image feature, wherein the twelfth image feature has the same size as the seventh image feature;

[0147] The twelfth image feature and the seventh image feature are fused through a first feature fusion layer to obtain a thirteenth image feature;

[0148] Inputting the thirteenth image feature into a twelfth network structure for convolution processing and feature enhancement processing to obtain a fourteenth image feature, wherein the twelfth network structure has the same network structure as the tenth network structure;

[0149] Inputting the fourteenth image feature into the thirteenth network structure for convolution processing and feature enhancement processing to obtain a fifteenth image feature, wherein the thirteenth network structure is the same as the eleventh network structure;

[0150] performing upsampling processing on the fifteenth image feature through a second upsampling module to obtain a sixteenth image feature, wherein the sixteenth image feature has the same size as the fifth image feature;

[0151] The fifth image feature and the sixteenth image feature are subjected to feature fusion by a second feature fusion layer to obtain a seventeenth image feature;

[0152] Inputting the seventeenth image feature into a fourteenth network structure for convolution processing and feature enhancement processing to obtain an eighteenth image feature; wherein the fourteenth network structure has the same network structure as the twelfth network structure;

[0153] Inputting the eighteenth image feature into a fifteenth network structure for convolution processing, feature enhancement processing and downsampling processing to obtain a nineteenth image feature, wherein the fifteenth network structure is the same as the thirteenth network structure;

[0154] The nineteenth image feature is fused with the fifteenth image feature through a third feature fusion layer to obtain a twentieth image feature;

[0155] Inputting the 20th image feature into a 16th network structure for convolution processing and feature enhancement processing to obtain a 21st image feature, wherein the 16th network structure is the same as the 14th network structure;

[0156] Inputting the twenty-first image feature into a seventeenth network structure for convolution processing, feature enhancement processing, and downsampling processing to obtain a twenty-second image feature, wherein the seventeenth network structure is the same as the fifteenth network structure;

[0157] The twenty-second image feature and the eleventh image feature are fused through a fourth feature fusion layer to obtain a twenty-third image feature;

[0158] Inputting the twenty-third image feature into an eighteenth network structure for convolution processing and feature enhancement processing to obtain a twenty-fourth image feature, wherein the eighteenth network structure is the same as the sixteenth network structure;

[0159] The twenty-fourth image feature, the twenty-first image feature and the eighteenth image feature are respectively input into the network head modules corresponding to their respective sizes for classification prediction, so as to obtain the shoulder key points and crotch key points of the target person.

[0160] In another aspect of the embodiments of the present application, an electronic device is provided, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0161] Memory, used to store computer programs;

[0162] The processor is used to implement any method step described in the first aspect of the embodiment of the present application when executing the program stored in the memory.

[0163] In another aspect of the embodiments of the present application, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, any method step described in the first aspect of the embodiments of the present application is implemented.

[0164] An embodiment of the present application also provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute any of the above-mentioned electronic device adjustment methods.

[0165] Beneficial effects of the embodiments of the present application:

[0166] The electronic device adjustment method, device, electronic device and storage medium provided in the embodiment of the present application can determine the shoulder key points and crotch key points corresponding to the target person by performing key point detection on the first image to be processed, calculate the shoulder-span image distance according to the coordinates of the shoulder key points and crotch key points in the image, and convert the shoulder-span image distance into a first distance representing the actual distance between the target person and the electronic device, so that the target wind speed can be determined according to the first distance, so that the electronic device can adjust the wind speed according to the distance between the user and the electronic device. Compared with the operation mode in the related art, the device adjustment method of the embodiment of the present application can be more convenient without the user having to perform key operations.

[0167] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0168] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0169] Figure 1 A flow chart of an electronic device adjustment method provided in an embodiment of the present application;

[0170] Figure 2 A schematic diagram of the network structure of the second neural network model or the third neural network model provided in an embodiment of the present application;

[0171] Figure 3 A schematic diagram of the network structure of the second network unit provided in an embodiment of the present application;

[0172] Figure 4 A schematic diagram of the network structure of the third network unit provided in an embodiment of the present application;

[0173] Figure 5 A schematic diagram of the network structure of a first neural network model provided in an embodiment of the present application;

[0174] Figure 6 A schematic diagram of a network structure of a first network structure provided in an embodiment of the present application;

[0175] Figure 7 A schematic diagram of a network structure of a third network structure provided in an embodiment of the present application;

[0176] Figure 8 A schematic diagram of a network structure of a residual structure provided in an embodiment of the present application;

[0177] Fig. 9 A schematic diagram of a network structure of a ninth network structure provided in an embodiment of the present application;

[0178] Fig.10 A schematic diagram of a network structure of a tenth network structure provided in an embodiment of the present application;

[0179] Fig.11 Another flow chart of the electronic device adjustment method provided in the embodiment of the present application;

[0180] Fig.12 A schematic diagram of a structure of an electronic device adjustment device provided in an embodiment of the present application;

[0181] Fig.13 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0182] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0183] As smart homes become increasingly popular, people have a growing demand for smart and convenient home appliances. In related technologies, people usually control devices by buttons, knobs, remote controls, etc. However, such control methods usually have certain limitations. For example, to adjust the device by buttons and knobs, the user needs to go to the device to operate; to adjust the device by remote control, the user needs to hold the remote control and aim it at the device to adjust it. This method is obviously not smart enough, so how to enable users to adjust the device more conveniently is an urgent problem to be solved.

[0184] In order to solve at least one of the above problems, a first aspect of an embodiment of the present application provides an electronic device adjustment method, the method comprising: Figure 1 Steps shown:

[0185] Step S101: Acquire a first image to be processed captured by a camera.

[0186] The first image to be processed includes a target person. The electronic device of the embodiment of the present application includes a fan and a camera. In one example, the electronic device may be an electric fan, and the camera is arranged on the electric fan to collect images of the environment in which the electric fan is located. In another example, the electronic device may be an air conditioner, wherein the air conditioner includes a fan structure, and the camera may be arranged on the air conditioner panel to collect images of the environment in which the air conditioner is located.

[0187] In practical applications, the camera can capture images periodically, for example, once every 0.1s, 0.5s or 1s, and perform image recognition on the captured images. If a person is recognized in the image, the image is determined to be the first image to be processed; if no person is recognized, the image is discarded or not processed.

[0188] Step S102: performing human body key point detection on the first image to be processed to determine the shoulder key points and crotch key points of the target person in the first image to be processed.

[0189] When there are multiple human objects in the first image to be processed, the target person can be determined by the pixel area of ​​each human object in the first image to be processed. In one example, contour detection is performed on the first image to be processed to obtain image areas corresponding to each human object, and the areas of the image areas corresponding to each human object are calculated, and the human object corresponding to the image area with the largest area is determined to be the target person.

[0190] When performing key point detection on the human body, key point detection can be performed directly through the pre-trained first neural network model. The first neural network model is a deep learning model for target detection. In one example, the target detection network model is a network model with a CSPNet (Cross Stage Partial Network, lightweight neural network structure) structure, OpenPose (a human pose estimation library based on deep learning), or a BlazePose (a lightweight convolutional neural network structure) neural network, a HRNet (High-Resolution Network, high-resolution network) structure neural network model, and other network models.

[0191] When training the first neural network model, it can be trained by using the collected data set or the data set in the related technology. In one example, the first neural network is trained by using the COCO (Common Objects in Context, a data set for image recognition) data set, so that the first neural network can detect 17 key points of the human body (nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, right ankle). In another example, when the first neural network model detects key points of the human body on the first image to be processed, it can classify the features of each pixel point and obtain the key point classification result of the pixel point. For example, through the first neural network model, the key point scores of a certain pixel point are obtained, and the key points of the target person are determined according to the key point scores, so as to select the shoulder key points and crotch key points of the target person from the key points.

[0192] The key point score is the confidence score of the pixel for each key point. In one example, taking the key point detection corresponding to the above 17 key points as an example, the scores of the above 17 key points corresponding to one pixel are (0.3, 0.1, 0.45, 0.14, 0.65, 0.9, 0.1, 0.26, 0.56, 0.14, 0.05, 0.02, 0.04, 0.03, 0.74, 0.03, 0.35). The scores of each key point are independent of each other. When filtering the key point scores, they can be filtered according to the preset threshold. In one example, 0.5 is selected as the threshold for filtering. The score of the nose key point is 0.9. If the score is greater than the threshold, it is considered that the key point can be used as the nose key point. In practical applications, key points can also be selected based on the target frame, where the first neural network model can predict the target frame of each pixel, calculate the intersection-and-union ratio between the target frame and the real frame, select the target frame with the largest intersection-and-union ratio, and use the target frame as the target frame corresponding to the final target person. For the key points in the target frame, the key points can be screened again based on the key point scores in the above example to determine the final shoulder key points and crotch key points.

[0193] Step S103: Calculate the shoulder-span image distance according to the image coordinates of the shoulder key points and the crotch key points in the first image to be processed, and convert the shoulder-span image distance into the first distance.

[0194] The first distance represents the distance between the target person and the electronic device. When converting the shoulder-to-shoulder image distance into the actual distance between the target person and the electronic device, a mathematical model for converting the two-dimensional image distance into the three-dimensional actual distance can be determined based on pre-calibrated camera internal and external parameters and other parameter information, thereby converting the shoulder-to-shoulder image distance into the first distance.

[0195] In practical applications, when determining the shoulder key points and hip key points of the target person in step S102, the coordinates of the shoulder key points and hip key points in the first image to be processed can be directly obtained, so that the shoulder-hip image distance can be calculated based on the coordinates of the shoulder key points and hip key points.

[0196] Step S104: query the preset wind speed correspondence, determine the target wind speed corresponding to the first distance, and adjust the fan to the target wind speed.

[0197] Among them, the wind speed correspondence is a preset correspondence between wind speed and distance. The wind speed correspondence can be set according to actual business needs. For example, different wind speed levels can be set according to different distance ranges. In an example, when the first distance is within the range of 8m-10m, the corresponding wind speed level is set to level 5, when the first distance is within the range of 6m-8m, the corresponding wind speed level is set to level 4; when the first distance is within the range of 4m-6m, the corresponding wind speed level is set to level 3; when the first distance is within the range of 2m-4m, the corresponding wind speed level is set to level 2; when the first distance is within the range of 0m-2m, the corresponding wind speed level is set to level 1.

[0198] By using the method of the embodiment of the present application, the shoulder key points and crotch key points corresponding to the target person can be determined by performing key point detection on the first image to be processed, and the shoulder-span image distance can be calculated according to the coordinates of the shoulder key points and crotch key points in the image, and the shoulder-span image distance can be converted into a first distance representing the actual distance between the target person and the electronic device, so that the target wind speed can be determined according to the first distance, so that the electronic device can adjust the wind speed according to the distance between the user and the electronic device. Compared with the operation method in the related art, the device adjustment method of the embodiment of the present application can be more convenient without the user having to perform key operations.

[0199] In a possible implementation, step S103 may calculate the first distance by the following steps:

[0200] Step 1: Calculate the coordinates of the shoulder key points and / or crotch key points by the following formula:

[0201] ;

[0202] ;

[0203] Among them, k x Indicates the horizontal coordinate of the shoulder key point or the crotch key point, k y The ordinate of the shoulder key point or the crotch key point is represented by the horizontal and vertical coordinates of the offset coordinates of the key point predicted by the first neural network model relative to the anchor point. kx and tky The horizontal and vertical coordinates of the anchor point coordinates are c x and c y .

[0204] The first neural network model performs downsampling on the first image to be processed to obtain downsampled image features, and takes the width and height of the downsampled image features to form a two-dimensional plane. Each coordinate point on the two-dimensional plane is an anchor point. The anchor point coordinates can be determined according to the size of the feature map. In an example, the size of the downsampled image feature is 20x20, and the anchor point coordinates in the downsampled image feature are (0, 0), (0, 1), (0, 2) ... (19, 19) in sequence.

[0205] Step 2: Calculate the coordinates of the center points of the two shoulder key points and the center points of the two hip key points using the following formula:

[0206] ;

[0207] Among them, k csx represents the horizontal coordinate of the center point of the two shoulder key points, k lsx Indicates the horizontal coordinate of the left shoulder key point of the target person, k rsx The horizontal coordinate of the key point of the right shoulder of the target person;

[0208] ;

[0209] Among them, k csy The ordinate of the center point of the two shoulder key points, k lsy Indicates the ordinate of the left shoulder key point of the target person, k rsy The vertical coordinate of the key point of the right shoulder of the target person;

[0210] ;

[0211] Among them, k chx represents the horizontal coordinate of the center point of the two cross-part key points, k lhx Indicates the horizontal coordinate of the left span key point of the target person, k rhx Indicates the horizontal coordinate of the right span key point of the target person;

[0212] ;

[0213] Among them, k chy Indicates the ordinate of the center point of the two cross-part key points, k lhy Indicates the ordinate of the left span key point of the target person, k rhy Indicates the ordinate of the right-span key point corresponding to the target;

[0214] Step 3: According to the coordinates of the center points of the two shoulder key points and the coordinates of the center points of the two span key points, the shoulder span image distance is calculated by the following formula:

[0215] ;

[0216] Among them, d pix represents the shoulder-span image distance;

[0217] Step 4: Convert the shoulder-span image distance to the first distance using the following formula:

[0218] ;

[0219] Where D represents the first distance, f represents the focal length of the camera, and d body Indicates the preset average shoulder span distance.

[0220] In one example, the electronic device can be set to different modes to distinguish users of different heights. For example, the electronic device is provided with an adult mode and a child mode. Then, the average shoulder span distance of the adult group can be counted as the preset average shoulder span distance when the electronic device works in the adult mode, and the average shoulder span distance of the child group can be counted as the preset average shoulder span distance when the electronic device works in the child mode.

[0221] By applying the method of the embodiment of the present application, the above formula can be used to realize the conversion of the target person's shoulder-span image distance to the first distance, so that the target wind speed can be determined according to the first distance, and then the wind speed can be adjusted according to the distance between the user and the electronic device, so that the user does not need to reach the device to press a button, and can adjust the electronic device more conveniently.

[0222] In a possible implementation, the electronic device may also recognize the gesture of the target person, so that the user can control the electronic device through the gesture, wherein step S102 may be implemented through the following steps:

[0223] The first image to be processed is input into a pre-trained first neural network model to obtain shoulder key points and crotch key points of the target person in the first image to be processed.

[0224] In practical applications, the first neural network model may also output a gesture frame of the target person, and the method of the embodiment of the present application may further include the following steps:

[0225] Step 1: According to the gesture frame, the first image to be processed is cut to obtain a gesture image.

[0226] In one example, when the first neural network model outputs the gesture frame of the target person, it can first predict the gesture frame for each pixel point, obtain the gesture frame corresponding to each pixel point and the gesture frame score, and select the gesture frame with the highest gesture frame score as the gesture frame of the target person, so as to extract the gesture image from the first image to be processed based on the gesture frame.

[0227] Step 2: Input the gesture image into the pre-trained second neural network model to obtain the gesture classification result corresponding to the gesture image.

[0228] The gesture classification result indicates the business gesture category corresponding to the gesture in the gesture image. The network structure of the second neural network model can be a neural network structure used for classification prediction in the related technology. In one example, the second neural network model can be a neural network with a CNN (Convolutional Neural Networks) structure, such as LeNet (a convolutional neural network), AlexNet (a neural network), VGG (Visual Geometry Group), and other network structures.

[0229] The output value of the second neural network model can be the same as the number of gesture categories. In an example, there are 5 categories of business gestures, and the second neural network model will output 5 values ​​for the gesture image. These 5 values ​​correspond to the classification scores of the gesture image for each business gesture, and the sum of these 5 values ​​is 1. For example, the output of the second neural network model is [0.1, 0.5, 0.2, 0.05, 0.15], where 0.5 has the largest score, and the value is 1 in the subscript position of the array, then it is determined that the business gesture category corresponding to the gesture image is category 1.

[0230] Among them, the business gesture is a pre-set gesture pattern. In another example, when recognizing a gesture image, the gesture in the gesture image can be directly matched with the pre-set gesture pattern in shape, and the business gesture pattern with the highest matching degree is selected as the gesture category corresponding to the gesture image.

[0231] Step 3: According to the gesture classification result, determine the device control instruction so that the fan performs the adjustment operation corresponding to the device control instruction.

[0232] Similarly, in actual applications, the correspondence between each business gesture category and the fan adjustment operation will be preset. For example, business gesture category 0 corresponds to increasing the wind speed level, business gesture category 1 corresponds to decreasing the wind speed level, business gesture category 2 corresponds to increasing the outlet temperature, and business gesture category 3 corresponds to decreasing the outlet temperature. In one example, the pre-set business gestures include aiming gestures, digital gestures, OK gestures, thumbs-up gestures, etc., and the business functions corresponding to each business gesture can be preset. For example, the aiming gesture corresponds to the switch of the fan. After the electronic device recognizes that the business gesture is an aiming gesture, it can execute the opposite state according to the current switch state of the fan. If the current state of the fan is the on state, the electronic device sends a shutdown command to the fan after detecting the aiming gesture; if the current state of the fan is the off state, the electronic device sends an on command to the fan after detecting the aiming gesture.

[0233] In another example, digital (1, 2, 3, 4, 5) gestures can be set as fan wind speed control gestures, corresponding to different wind speed level control instructions of the fan. In another example, the OK gesture corresponds to the target tracking function of the fan. After the electronic device recognizes that the business gesture is an OK gesture, it sends a target tracking instruction to the fan so that the fan can automatically adjust the wind speed and direction according to the user's distance. In actual applications, business gestures and their corresponding functions may include but are not limited to the above-mentioned business gestures and their corresponding functions. For example, business gestures and their corresponding functions may also include a vertical eight gesture corresponding to a fan fixed wind direction blowing function, a thumbs-up gesture corresponding to a fan natural wind function, and a thumbs-up gesture corresponding to a fan sleep wind function.

[0234] By using the method of the embodiment of the present application, a gesture image can be obtained from the image to be processed through the gesture frame, thereby performing gesture detection on the gesture image, obtaining a gesture classification result, and determining the relevant adjustment operation of the fan according to the gesture classification result. Furthermore, the user can directly control the electronic device through gestures. Compared with the prior art, the method of the embodiment of the present application does not require the user to go to the device to control it, nor does it require the user to find a remote control or other control device to control the electronic device, thereby realizing a more convenient way to control the electronic device.

[0235] In a possible implementation, since the gesture image is captured from the first image to be processed, its clarity may not be able to meet the requirement that the second neural network model can clearly identify the gesture category. Therefore, before inputting the gesture image into the second neural network model in step 2, the embodiment of the present application can also perform a quality inspection on the gesture image. In one example, the quality inspection of the gesture image can be performed by the following steps:

[0236] Step A: Input the gesture image into a pre-trained third neural network model to obtain a clarity score and a blur score of the gesture image.

[0237] Step B: If the blur score is greater than or equal to the clarity score, the gesture image is discarded.

[0238] Step C: When the clarity score is greater than the blur score, execute step 2.

[0239] The third neural network model is used to detect the clarity of the gesture graph. The network structure of the third neural network model can be the same as or different from the network structure of the second neural network model. In one example, the third neural network model and the second neural network model are both neural networks with CNN structures. In practical applications, the third neural network model has been pre-trained through a training set and a test set, wherein both the training set and the test set include clear photos and blurred photos.

[0240] When the third neural network model classifies and predicts the gesture image, it will obtain the fuzziness score and clarity score of the gesture image respectively. When the fuzziness score is greater than or equal to the clarity score, it is considered that the clarity of the gesture image is not enough for the second neural network model to clearly identify the business gesture category corresponding to the gesture in the gesture image, and the gesture image can be discarded or not processed. When the clarity score of the gesture image is greater than the fuzziness score, it is considered that the gesture image is relatively clear, so step 2 is executed to input it into the second neural network model.

[0241] In one example, the network structure of the second neural network model and the network structure of the third neural network model are both Figure 2 The network structure shown. The gesture image (gesture image of size 128x128x3) is input into the second neural network model or the third neural network model. The first network unit, the second network unit and multiple third network units perform feature extraction, convolution, feature enhancement and other processing on the gesture image to obtain the image features corresponding to the gesture image. The image features are then classified and predicted through the pooling layer and the fully connected layer to obtain the category of the gesture image for output.

[0242] The first network unit includes a convolution layer, a batch normalization layer, and an activation function layer. The network structure of the second network unit is as follows: Figure 3 As shown, it includes a network module with the same structure as the first network unit, a fourth network unit, a feature fusion layer and an activation function layer. Among them, the fourth network unit includes a convolution layer and a batch normalization layer. The network structure of the third network unit is as shown Figure 4 As shown, compared with the second network unit, the third network unit also includes a network module with the same structure as the fourth network unit, so that more image features of the gesture image can be obtained.

[0243] By applying the method of the embodiment of the present application, the gesture images are input into the third neural network model for clarity detection before being input into the second neural network model for gesture detection, thereby ensuring that the gesture images input into the second neural network model are clear images, facilitating gesture detection by the second neural network model, and avoiding the waste of computing power caused by gesture detection on blurred images.

[0244] In a possible implementation, the embodiment of the present application may also control the direction of the fan according to the position of the target person. In one example, after step S101, the embodiment of the present application may further include the following steps:

[0245] Step (1): Obtain a second image to be processed collected by the fan.

[0246] The second image to be processed is an image captured after the first image to be processed. In practical applications, the camera captures images at a certain interval. In an example, the camera captures an image every 5 seconds. At 10 seconds, the camera will capture two images, and the first captured image will be determined as the first image to be processed, and the later captured image will be determined as the second image to be processed.

[0247] Step (ii): input the first image to be processed and the second image to be processed into the first neural network model respectively to obtain the first target frame and the second target frame.

[0248] Among them, the first target frame represents the position of the target person in the first image to be processed, and the second target frame represents the position of the target person in the second image to be processed. In practical applications, the first neural network model outputs the target frame while outputting the key points and gesture frames. The target frame includes the coordinates and width and height of the target frame. The position of the target frame can be calculated based on the width and height of the predicted frame predicted by the first neural network model, the offset coordinates of the predicted frame relative to the anchor point, the anchor point coordinates, and the downsampling ratios of the first image to be processed and the second image to be processed.

[0249] In one example, the first neural network model downsamples the first image to be processed or the second image to be processed to obtain a two-dimensional image feature, and each coordinate point on the two-dimensional image feature is an anchor point. The first neural network model will output a set of prediction values ​​for each anchor point, and the prediction value includes the offset coordinates of the prediction box relative to the anchor point, the width of the prediction box, the height of the prediction box, and the score of the target box. Among them, the score of the target box is the confidence score of the pixel point corresponding to the target person at the anchor point. The set of prediction values ​​with the highest score for the target box is selected to calculate the target box.

[0250] In an example, the target box can be calculated by the following formula:

[0251] ;

[0252] ;

[0253] Among them, the coordinates of the target box are (b x , b y ), the offset coordinates of the predicted box relative to the anchor point are (t x , t y ), the coordinates of the anchor point are (c x , c y ), strides represents the downsampling scaling ratio.

[0254] ;

[0255] ;

[0256] Among them, the width and height of the target box are (b w , b h ), the width and height of the predicted target box are (t w , t h ).

[0257] Step (three): Calculate the third target frame of the target person based on the first target frame.

[0258] The third target frame indicates the predicted position of the target person in the second image to be processed. When predicting the third target frame, the prediction and tracking of the target person can be achieved through a Kalman filter. In another example, the prediction and tracking of the target person can also be achieved through particle filtering, set membership filtering, etc.

[0259] Step (iv): Match the positional relationship between the second target frame and the third target frame to obtain a matching result.

[0260] In the embodiment of the present application, the second target frame represents the position of the target person in the second image to be processed in the actually captured image, and the third target frame represents the predicted position of the target person in the second image to be processed. The position relationship matching of the second target frame and the third target frame can be determined by calculating the intersection-and-union ratio between the second target frame and the third target frame.

[0261] In an example, the positional relationship between the second target frame and the third target frame can be matched by the following steps:

[0262] Step a: Calculate the intersection-over-union ratio of the second target frame and the third target frame.

[0263] Step b: when the intersection-over-union ratio is greater than or equal to a preset threshold, determining that the matching result indicates that the target person in the first image to be processed and the target person in the second image to be processed are the same target person.

[0264] Among them, the intersection and union of the second target frame and the third target frame can be calculated according to the coordinates and width and height of the second target frame and the third target frame respectively, so as to calculate the ratio of the intersection and the union to obtain the intersection-union ratio of the second target frame and the third target frame.

[0265] The preset threshold can be determined according to actual business needs. The larger the preset threshold is set, the higher the credibility of determining the same target person. However, if the target person moves faster, the target person in the first image to be processed and the target person in the second image to be processed may be misjudged as different persons. In an example, the preset threshold is set to 0.5. If the intersection-over-union ratio is greater than or equal to 0.5, it is considered that the second target frame and the third target frame can be associated, thereby determining that the target person in the first image to be processed and the target person in the second image to be processed are the same target person.

[0266] By applying the method of the embodiment of the present application, position matching is performed by calculating the interaction ratio between the second target frame and the third target frame, so that the position of the target person in the captured second image to be processed can be associated with the predicted position of the target person in the second image to be processed. If there is a correlation, it is determined that the first image to be processed and the second image to be processed are images of the same target person, thereby achieving tracking of the target person, and then the device adjustment instructions can be determined based on the position change information of the target person.

[0267] Step (V): When the matching result indicates that the target person in the first image to be processed and the target person in the second image to be processed are the same target person, the difference between the first target frame and the second target frame is calculated to obtain position transformation information.

[0268] Step (six): Determine the device adjustment instruction according to the position change information so that the fan is adjusted according to the device adjustment instruction.

[0269] After determining that the target person in the first image to be processed and the target person in the second image to be processed are the same target person, the position transformation information can be calculated based on the first target frame and the second target frame obtained in step (ii). The position transformation information can be obtained by calculating the difference between the first target frame and the second target frame to obtain the displacement transformation of the target person in the image, thereby converting the displacement transformation from two-dimensional information to three-dimensional position transformation information. Similarly, the mathematical model of two-dimensional and three-dimensional conversion can be determined based on the parameters pre-calibrated by the camera, and the two-dimensional displacement transformation can be converted into three-dimensional position transformation information through the mathematical model.

[0270] In practical applications, in order to avoid the accumulation of errors, the two-dimensional position information of the target person in the image can be obtained in real time based on the image information collected by the camera, and the two-dimensional position information can be directly converted into three-dimensional position information through a predetermined two-dimensional and three-dimensional conversion mathematical model, so as to determine the device adjustment instructions based on the three-dimensional position information. Among them, the device adjustment instructions include but are not limited to the instructions for controlling the fan to change the wind direction, the instructions for controlling the fan to change the wind speed, the instructions for controlling the fan to change the air outlet temperature, and other adjustment instructions.

[0271] By applying the method of the embodiment of the present application, the target person can be tracked by matching the position relationship between the second target frame and the third target frame, so that the device adjustment instructions can be determined based on the position change information of the target person, so that the fan of the electronic device can supply air to the user's location more intelligently, reduce useless air supply, and avoid energy waste.

[0272] In a possible implementation, when predicting the third target frame, the prediction may be performed by a Kalman filter method, as follows:

[0273] Step (a): Calculate the initial state vector of the target person in the first image to be processed according to the first target frame.

[0274] Step (b): Calculate the error between the first predicted state vector and the first predicted state vector according to the initial state vector.

[0275] Step (c): Calculate the measurement value of the target person in the second image to be detected according to the measurement matrix of the measuring device.

[0276] Step (d): In combination with the measured value, the first predicted state vector is corrected to obtain a first corrected state vector.

[0277] Step (e): Calculate the third target frame based on the first corrected state vector.

[0278] The state vector is used to represent the motion state and motion position of the target person, including but not limited to the coordinates of the target person corresponding to the target frame in the image, the width and height of the target frame, the motion data of the target person (speed, direction), the probability distribution of the target person's motion, etc. In one example, steps (a) to (e) of the embodiment of the present application can be directly implemented by a Kalman filter algorithm.

[0279] The state vector corresponding to the first target frame is used as the initial state vector, and the first predicted state vector is calculated through the prediction model of the Kalman filter. In an example, the first predicted state vector is calculated by the following formula:

[0280] ;

[0281] Among them, x k1 represents the first predicted state vector, A k represents the state transfer matrix, x k-1 represents the initial state vector, B k represents the control matrix, u k represents the control vector, w k Represents the Gaussian noise of the prediction process.

[0282] In practical applications, there is a prediction error between the predicted value and the actual value, which can be calculated by the following formula:

[0283] ;

[0284] Among them, P k1 represents the prediction error corresponding to the first prediction state vector, P k-1 represents the prediction error corresponding to the initial state vector; Q k represents the noise covariance matrix, Represents the transpose operation on the state transfer matrix.

[0285] In Kalman filtering, the measured values ​​obtained by the measuring device are also included, where the measuring device can be a sensor, a camera and other devices. The measured values ​​can be calculated in the following ways:

[0286] ;

[0287] Among them, z k Indicates the measured value, H k represents the measurement matrix detected by the measuring device, v k represents the Gaussian noise of the observation process.

[0288] Correspondingly, there is also a measurement error y when the measuring device is measuring k , the measurement error can be obtained by v k If , the measurement covariance can be calculated by the following formula:

[0289] ;

[0290] Among them, S k represents the measurement covariance, R k represents the measurement noise covariance matrix, Represents the transpose operation on the measurement matrix.

[0291] Based on the prediction error and measurement covariance, the Kalman gain can be calculated by the following formula:

[0292] ;

[0293] Among them, K k represents the Kalman gain.

[0294] The first predicted state vector is corrected by the following formula:

[0295] ;

[0296] Among them, x k2 represents the first revised state vector.

[0297] Correspondingly, the prediction error is also corrected:

[0298] ;

[0299] Among them, P K2 represents the corrected forecast error.

[0300] After correcting the first predicted state vector to obtain the first corrected state vector, the Kalman filter algorithm can also realize the conversion between the state vector and the target frame to obtain the third target frame.

[0301] By applying the method of the embodiment of the present application, the target person can be tracked through the Kalman filtering method, so that the device adjustment instructions can be determined according to the position change information of the target person, so that the fan of the electronic device can supply air to the user's location more intelligently, reduce useless air supply, and avoid energy waste.

[0302] In a possible implementation, the network structure of the first neural network model is as follows: Figure 5 As shown, taking the step of predicting the first image to be processed by the first neural network model to obtain the shoulder key points and crotch key points of the target person as an example, the processing process of the first neural network model on the first image to be processed is as follows:

[0303] Inputting the first image to be processed into the first neural network model, and performing feature extraction and downsampling processing on the first image to be processed by the first network structure in the first neural network model to obtain first image features;

[0304] Inputting the first image feature into the second network structure for convolution processing, feature enhancement processing and downsampling processing to obtain the second image feature; wherein the second network structure is the same as the first network structure;

[0305] Inputting the second image feature into the third network structure for feature enhancement processing to obtain a third image feature;

[0306] Inputting the third image feature into a fourth network structure for convolution processing, feature enhancement processing and downsampling processing to obtain a fourth image feature; wherein the fourth network structure is the same as the second network structure;

[0307] Inputting the fourth image feature into the fifth network structure for feature enhancement processing to obtain a fifth image feature; wherein the fifth network structure is the same as the third network structure;

[0308] Inputting the fifth image feature into a sixth network structure for convolution processing, feature enhancement processing, and downsampling processing to obtain a sixth image feature, wherein the sixth network structure is the same as the fourth network structure;

[0309] Inputting the sixth image feature into the seventh network structure for feature enhancement processing to obtain a seventh image feature, wherein the seventh network structure has the same network structure as the fifth network structure;

[0310] Inputting the seventh image feature into the eighth network structure for convolution processing, feature enhancement processing and downsampling processing to obtain an eighth image feature; wherein the eighth network structure is the same as the sixth network structure;

[0311] Inputting the eighth image feature into the ninth network structure for multi-scale pooling operation, fusing image features of different scales, and obtaining a ninth image feature;

[0312] Inputting the ninth image feature into the tenth network structure for convolution processing and feature enhancement processing to obtain a tenth image feature, wherein the tenth image feature, the seventh image feature, and the fifth image feature have different sizes;

[0313] Performing convolution processing and feature enhancement processing on the tenth image feature through an eleventh network structure to obtain an eleventh image feature, wherein the eleventh network structure is the same as the eighth network structure;

[0314] Performing upsampling processing on the eleventh image feature through a first upsampling module to obtain a twelfth image feature, wherein the twelfth image feature has the same size as the seventh image feature;

[0315] The twelfth image feature and the seventh image feature are fused through a first feature fusion layer to obtain a thirteenth image feature;

[0316] Inputting the thirteenth image feature into the twelfth network structure for convolution processing and feature enhancement processing to obtain the fourteenth image feature, wherein the twelfth network structure has the same network structure as the tenth network structure;

[0317] Inputting the fourteenth image feature into the thirteenth network structure for convolution processing and feature enhancement processing to obtain the fifteenth image feature, wherein the thirteenth network structure has the same network structure as the eleventh network structure;

[0318] Upsampling the fifteenth image feature by a second upsampling module to obtain a sixteenth image feature, wherein the sixteenth image feature has the same size as the fifth image feature;

[0319] The fifth image feature and the sixteenth image feature are fused through a second feature fusion layer to obtain a seventeenth image feature;

[0320] Inputting the seventeenth image feature into the fourteenth network structure for convolution processing and feature enhancement processing to obtain the eighteenth image feature; wherein the fourteenth network structure has the same network structure as the twelfth network structure;

[0321] Inputting the eighteenth image feature into the fifteenth network structure for convolution processing, feature enhancement processing and down-sampling processing to obtain the nineteenth image feature, wherein the fifteenth network structure has the same network structure as the thirteenth network structure;

[0322] The nineteenth image feature is fused with the fifteenth image feature through a third feature fusion layer to obtain a twentieth image feature;

[0323] Inputting the twentieth image feature into the sixteenth network structure for convolution processing and feature enhancement processing to obtain the twenty-first image feature, wherein the sixteenth network structure has the same network structure as the fourteenth network structure;

[0324] Inputting the twenty-first image feature into the seventeenth network structure for convolution processing, feature enhancement processing and down-sampling processing to obtain the twenty-second image feature, wherein the seventeenth network structure is the same as the fifteenth network structure;

[0325] The twenty-second image feature and the eleventh image feature are fused through a fourth feature fusion layer to obtain a twenty-third image feature;

[0326] Inputting the twenty-third image feature into the eighteenth network structure for convolution processing and feature enhancement processing to obtain the twenty-fourth image feature, wherein the eighteenth network structure has the same network structure as the sixteenth network structure;

[0327] The twenty-fourth image feature, the twenty-first image feature and the eighteenth image feature are respectively input into the network head modules corresponding to their respective sizes for classification prediction, so as to obtain the shoulder key points and crotch key points of the target person.

[0328] Among them, although the network modules with the same network structure have the same structure, their parameters can be set to different network parameters according to their corresponding image processing operations. In an example, although the network structures of the thirteenth network structure and the fifteenth network structure are the same, the downsampling parameter of the fifteenth network structure is 2, and the downsampling parameter of the thirteenth network structure is 1.

[0329] The network head module is the classification prediction module of the first neural network, and each network head module corresponds to a different image size. Figure 5 As shown, after multiple downsamplings, the first image to be processed (640x640x3) obtains three image features of different sizes (the twenty-fourth image feature 20x20x512, the twenty-first image feature 40x40x256, and the eighteenth image feature 80x80x128). The neural network includes three network head modules corresponding to different downsampling depths. The network structure of each network head module is the same, but due to different downsampling depths, its parameters may be different. Specifically, the parameters of the network structure are set or trained according to actual business needs.

[0330] The network head module obtains the corresponding prediction result by performing convolution and feature enhancement on the image features. Taking the network head module corresponding to the size of "20x20x512" as an example, the network head module includes multiple CBR (Conv+ Batch Norm+ Relu, convolution layer + batch normalization layer + activation function layer) modules that are the same as the first network structure. Through multiple convolution layers, the classification results of different channels (20x20x2, 20x20x1, 20x20x4, 20x20x51) can be output. The classification results of each convolution layer are then fused through the feature fusion layer to obtain the classification result (20x20x58) corresponding to the downsampled image feature of this size and output it (output 2). It should be noted that the first network structure in the network head module only represents the CBR module with the same network structure as the first network structure, and its parameters do not need to be the same as the first network structure, and the parameters of each convolution layer and the feature fusion layer are not necessarily the same.

[0331] In practical applications, the output of the network head module includes multiple categories and scores of each category. Taking the network head module corresponding to the size of "20x20x512" as an example, its output (20x20x58) can include 58 values, namely, the x (x coordinate offset of the predicted point relative to the anchor point), y (y coordinate offset of the predicted point relative to the anchor point), w (width of the predicted target box), h (height of the predicted target box), obj (score of the target box), cls1 (category 1 score), cls2 (category 2 score), kx1 (x coordinate offset of key point 1), ky1 (y coordinate offset of key point 1), ks1 (score of key point 1), kx2 (x coordinate offset of key point 2), ky2 (y coordinate offset of key point 1), ks2 (score of key point 2)..., kx17 (x coordinate offset of key point 17), ky17 (y coordinate offset of key point 17), ks17 (score of key point 17). In practical applications, the target frame can be screened by predicting the intersection-over-union ratio of the target frame to determine the target frame corresponding to the target person. Then, the key points in the target frame are screened according to the scores of each category, and finally the target frame, gesture frame and key points corresponding to the target person in the first image to be processed are obtained.

[0332] Among them, the first network structure is a CBR module, such as Figure 6 As shown in , the network structure includes a convolution layer, a batch normalization layer, and an activation function layer. The third network structure is as follows Figure 7 As shown in the figure, similar to the CSPNet (Cross Stage Partial Network) network structure, this network structure is the CSP-X (network structure applied to the Backbone part of CSPNet) structure, including multiple CBR modules, residual structures and feature fusion layers that are the same as the first network structure. The original input can be divided into two branches, and convolution operations are performed on each branch to halve the number of channels. Then, one branch performs a Bottleneck XN (residual connection) operation, and then the two branches are feature fused, so that the input and output of the third network structure are the same size, so that the network can be built deeper and obtain more image information. In an example, the residual structure is Figure 8 The network structure shown includes multiple CBR modules and feature fusion layers with the same structure as the first network structure, so that the output of the network layer can be connected to the output across multiple network layers to avoid the disappearance or explosion of the model gradient.

[0333] The ninth network structure is as follows Fig. 9As shown in Figure 1, similar to the CSPNet network structure, this network structure is an SPP structure (Spatial Pyramid Pooling), which includes multiple CBR modules, maximum pooling layers, and feature fusion layers that are the same as the first network structure. Fig.10 As shown, similar to the CSPNet network structure, this structure is a CSP2-X (network structure applied to the Neck (middle connection) part of CSPNet) structure, including multiple CBR modules and feature fusion layers that are the same as the first network structure.

[0334] By applying the method of the embodiment of the present application, the target frame, gesture frame and key points corresponding to the target person can be detected through the first neural network, so that the position information and / or gesture information of the target person can be calculated based on the target frame, gesture frame and key points, and then the electronic device can be adjusted according to the position information and / or gesture information of the target person, providing users with a more convenient human-computer interaction method and improving the user experience.

[0335] In a possible implementation, the present application may be implemented by Fig.11 The device is adjusted according to the flowchart shown in the figure. First, the target detection network performs target frame detection, gesture frame detection, and key point detection on the image to be processed to obtain the target frame, gesture frame, and key point corresponding to the target person in the image to be processed. The target detection network is the first neural network model pre-trained in the above embodiment.

[0336] After obtaining the target frame, gesture frame and key points, they can also be screened according to the confidence scores of each target frame, each gesture frame and each key point, filtering out the target frames, gesture frames and key points whose confidence scores do not meet the preset score threshold, and determining the target frames, gesture frames and key points that represent the information related to the target person.

[0337] After determining the target frame and key points, the target person can be tracked by Kalman filtering. The target person can be determined based on the pixel area of ​​each person object in the image to be detected, for example, the person object with the largest pixel area is determined as the target person.

[0338] After selecting the target person, the first distance is calculated according to the key point information of the target person, so that the wind speed control instruction can be determined according to the first distance. The target person can be tracked by Kalman filtering. When it is determined that the images collected before and after are the same target person, the position change information of the target person can be calculated according to the collected images, and the adjustment instruction of the electronic device can be determined according to the position change information and the first distance.

[0339] When performing gesture recognition of a user, it is necessary to first extract the gesture graph corresponding to the gesture from the image to be processed based on the gesture frame of the target person obtained by the target detection network, and recognize the gesture in the gesture graph. Before performing gesture recognition, the gesture graph can be quality-checked, for example, the gesture graph is sent to a quality filtering model to obtain the clarity score and blur score of the gesture graph. The quality filtering model is the pre-trained second neural network model mentioned above.

[0340] The clarity score and the blur score are used to determine whether the gesture graph is clear. If not, the gesture graph is not processed and the judgment of the gesture graph is terminated directly. If so, the gesture graph is input into the gesture prediction model for gesture prediction, wherein the gesture prediction model is the third neural network model pre-trained in the above embodiment. According to the gesture classification result of the gesture graph, it is determined whether the gesture in the gesture graph is a pre-set business gesture. If so, the corresponding gesture adjustment instruction is determined according to the gesture type; if not, the gesture graph is not processed and the judgment of the gesture graph is terminated.

[0341] In actual applications, the electronic device can also maintain the device adjustment method in the relevant technology. For example, the electronic device is provided with a key module, and the user can directly press a key to trigger the corresponding device adjustment instruction. Among them, there can be a priority between the device adjustment instructions triggered by different methods. For example, during the use of the electronic device, the electronic device adjusts the device in real time according to the user's location information, but if the user triggers the gesture adjustment instruction through a gesture at this time, the electronic device will prioritize the execution of the relevant adjustment operation according to the gesture adjustment instruction.

[0342] By applying the method of the embodiment of the present application, the position information and / or gesture information of the target person can be calculated based on the target frame, gesture frame and key points, and then the electronic device can be adjusted based on the position information and / or gesture information of the target person, providing users with a more convenient human-computer interaction method and improving the user experience.

[0343] According to a second aspect of the present application, there is provided an electronic device adjustment device, wherein the electronic device includes a fan and a camera. The device includes: Fig.12 The structure shown:

[0344] The first image to be processed acquisition module 1201 is used to acquire a first image to be processed captured by a camera; wherein the first image to be processed includes a target person;

[0345] The key point detection module 1202 is used to perform human key point detection on the first image to be processed, and determine the shoulder key points and crotch key points of the target person in the first image to be processed;

[0346] A first distance calculation module 1203 is used to calculate the shoulder-span image distance according to the image coordinates of the shoulder key points and the crotch key points in the first image to be processed, and convert the shoulder-span image distance into a first distance, wherein the first distance represents the distance between the target person and the electronic device;

[0347] The target wind speed determination module 1204 is used to query the preset wind speed correspondence, determine the target wind speed corresponding to the first distance, and adjust the fan to the target wind speed, wherein the wind speed correspondence is a preset correspondence between wind speed and distance.

[0348] In a possible implementation, the first distance calculation module includes:

[0349] The key point coordinate calculation submodule is specifically used to calculate the coordinates of the shoulder key points and / or the crotch key points through the following formula:

[0350] ;

[0351] ;

[0352] Among them, k x Indicates the horizontal coordinate of the shoulder key point or the crotch key point, k y The ordinate of the shoulder key point or the crotch key point is represented by the horizontal and vertical coordinates of the offset coordinates of the key point predicted by the first neural network model relative to the anchor point. kx and t ky The horizontal and vertical coordinates of the anchor point coordinates are c x and c y ; strides represents the downsampling scaling ratio;

[0353] The center point coordinate calculation submodule is specifically used to calculate the coordinates of the center points of the two shoulder key points and the coordinates of the center points of the two span key points through the following formula:

[0354] ;

[0355] Among them, k csx represents the horizontal coordinate of the center point of the two shoulder key points, k lsx Indicates the horizontal coordinate of the left shoulder key point of the target person, k rsx The horizontal coordinate of the key point of the right shoulder of the target person;

[0356] ;

[0357] Among them, k csy The ordinate of the center point of the two shoulder key points, k lsy Indicates the ordinate of the left shoulder key point of the target person, k rsyThe vertical coordinate of the key point of the right shoulder of the target person;

[0358] ;

[0359] Among them, k chx represents the horizontal coordinate of the center point of the two cross-part key points, k lhx Indicates the horizontal coordinate of the left span key point of the target person, k rhx Indicates the horizontal coordinate of the right span key point of the target person;

[0360] ;

[0361] Among them, k chy Indicates the ordinate of the center point of the two cross-part key points, k lhy Indicates the ordinate of the left span key point of the target person, k rhy Indicates the ordinate of the right-span key point corresponding to the target;

[0362] The shoulder-span image distance calculation submodule is specifically used to calculate the shoulder-span image distance according to the coordinates of the center points of the two shoulder key points and the coordinates of the center points of the two span key points by the following formula;

[0363] ;

[0364] Among them, d pix represents the shoulder-span image distance;

[0365] A first distance calculation submodule is specifically used to convert the shoulder span image distance into a first distance through the following formula;

[0366] ;

[0367] Where D represents the first distance, f represents the focal length of the camera, and d body Indicates the preset average shoulder span distance.

[0368] In a possible implementation, the key point detection module includes:

[0369] A first neural network model prediction submodule is specifically used to input the first image to be processed into a pre-trained first neural network model to obtain the shoulder key points and crotch key points of the target person in the first image to be processed;

[0370] The first neural network model further outputs a gesture frame of the target person, and the device further includes:

[0371] An image cutting module, used for cutting the first image to be processed according to the gesture frame to obtain a gesture image;

[0372] A gesture detection module, used to input the gesture image into a pre-trained second neural network model to obtain a gesture classification result corresponding to the gesture image, wherein the gesture classification result indicates a business gesture category corresponding to the gesture in the gesture image;

[0373] The device control instruction determination module is used to determine the device control instruction according to the gesture classification result so that the fan executes the adjustment operation corresponding to the device control instruction.

[0374] In a possible implementation, the device further includes:

[0375] A clarity detection module is used to input the gesture image into a pre-trained third neural network model to obtain a clarity score and a blur score of the gesture image; wherein the third neural network model is used to perform clarity detection on the gesture image;

[0376] A fuzzy image processing module, used for discarding the gesture image when the fuzziness score is greater than or equal to the clarity score;

[0377] The clear image processing module is used to execute the steps of inputting the gesture image into a pre-trained second neural network model to obtain a gesture classification result corresponding to the gesture image when the clarity score is greater than the blur score.

[0378] In a possible implementation, the device further includes:

[0379] A second image to be processed acquisition module, used to acquire a second image to be processed acquired by the fan, wherein the second image to be processed is an image acquired after the first image to be processed;

[0380] A target frame acquisition module is used to input the first image to be processed and the second image to be processed into the first neural network model respectively to obtain a first target frame and a second target frame, wherein the first target frame represents the position of the target person in the first image to be processed, and the second target frame represents the position of the target person in the second image to be processed;

[0381] A third target frame prediction module is used to calculate a third target frame of the target person according to the first target frame, wherein the third target frame represents a predicted position of the target person in the second image to be processed;

[0382] A position relationship matching module is used to match the position relationship between the second target frame and the third target frame to obtain a matching result;

[0383] A position transformation information calculation module, used to calculate the difference between the first target frame and the second target frame to obtain position transformation information when the matching result indicates that the target person in the first image to be processed and the target person in the second image to be processed are the same target person;

[0384] The device adjustment instruction determination module is used to determine the device adjustment instruction according to the position change information so that the fan is adjusted according to the device adjustment instruction.

[0385] In a possible implementation, the third target box prediction module includes:

[0386] An initial state vector calculation submodule, specifically used to calculate the initial state vector of the target person in the first image to be processed according to the first target frame;

[0387] A prediction vector calculation submodule, specifically used to calculate the error between the first prediction state vector and the second prediction state vector according to the initial state vector;

[0388] The measurement value calculation submodule is specifically used to calculate the measurement value of the target person in the second image to be detected and the error between the measurement values ​​according to the measurement matrix of the measuring device;

[0389] A Kalman gain calculation submodule, specifically used to calculate the Kalman gain by combining the error of the measured value with the error of the first predicted state vector;

[0390] The prediction vector correction submodule is specifically used to correct the first prediction state vector through the Kalman gain to obtain a first corrected state vector;

[0391] The third target frame calculation submodule is specifically used to calculate the third target frame according to the first corrected state vector.

[0392] In a possible implementation, the position relationship matching module includes:

[0393] The intersection-and-union ratio calculation submodule is specifically used to calculate the intersection-and-union ratio of the second target frame and the third target frame;

[0394] The intersection-and-union ratio judgment submodule is specifically used to determine that the matching result indicates that the target person in the first image to be processed and the target person in the second image to be processed are the same target person when the intersection-and-union ratio is greater than or equal to a preset threshold.

[0395] In a possible implementation, the key point detection module includes:

[0396] The image processing submodule is specifically used to input the first image to be processed into the first neural network model, and the first network structure in the first neural network model performs feature extraction and downsampling processing on the first image to be processed to obtain first image features;

[0397] Inputting the first image feature into the second network structure for convolution processing, feature enhancement processing and downsampling processing to obtain the second image feature; wherein the second network structure is the same as the first network structure;

[0398] Inputting the second image feature into the third network structure for feature enhancement processing to obtain a third image feature;

[0399] Inputting the third image feature into a fourth network structure for convolution processing, feature enhancement processing and downsampling processing to obtain a fourth image feature; wherein the fourth network structure is the same as the second network structure;

[0400] Inputting the fourth image feature into the fifth network structure for feature enhancement processing to obtain a fifth image feature; wherein the fifth network structure is the same as the third network structure;

[0401] Inputting the fifth image feature into a sixth network structure for convolution processing, feature enhancement processing, and downsampling processing to obtain a sixth image feature, wherein the sixth network structure is the same as the fourth network structure;

[0402] Inputting the sixth image feature into the seventh network structure for feature enhancement processing to obtain a seventh image feature, wherein the seventh network structure has the same network structure as the fifth network structure;

[0403] Inputting the seventh image feature into the eighth network structure for convolution processing, feature enhancement processing and downsampling processing to obtain an eighth image feature; wherein the eighth network structure is the same as the sixth network structure;

[0404] Inputting the eighth image feature into the ninth network structure for multi-scale pooling operation, fusing image features of different scales, and obtaining a ninth image feature;

[0405] Inputting the ninth image feature into the tenth network structure for convolution processing and feature enhancement processing to obtain a tenth image feature, wherein the tenth image feature, the seventh image feature, and the fifth image feature have different sizes;

[0406] Performing convolution processing and feature enhancement processing on the tenth image feature through an eleventh network structure to obtain an eleventh image feature, wherein the eleventh network structure is the same as the eighth network structure;

[0407] Performing upsampling processing on the eleventh image feature through a first upsampling module to obtain a twelfth image feature, wherein the twelfth image feature has the same size as the seventh image feature;

[0408] The twelfth image feature and the seventh image feature are fused through a first feature fusion layer to obtain a thirteenth image feature;

[0409] Inputting the thirteenth image feature into the twelfth network structure for convolution processing and feature enhancement processing to obtain the fourteenth image feature, wherein the twelfth network structure has the same network structure as the tenth network structure;

[0410] Inputting the fourteenth image feature into the thirteenth network structure for convolution processing and feature enhancement processing to obtain the fifteenth image feature, wherein the thirteenth network structure has the same network structure as the eleventh network structure;

[0411] Upsampling the fifteenth image feature by a second upsampling module to obtain a sixteenth image feature, wherein the sixteenth image feature has the same size as the fifth image feature;

[0412] The fifth image feature and the sixteenth image feature are fused through a second feature fusion layer to obtain a seventeenth image feature;

[0413] Inputting the seventeenth image feature into the fourteenth network structure for convolution processing and feature enhancement processing to obtain the eighteenth image feature; wherein the fourteenth network structure has the same network structure as the twelfth network structure;

[0414] Inputting the eighteenth image feature into the fifteenth network structure for convolution processing, feature enhancement processing and down-sampling processing to obtain the nineteenth image feature, wherein the fifteenth network structure has the same network structure as the thirteenth network structure;

[0415] The nineteenth image feature is fused with the fifteenth image feature through a third feature fusion layer to obtain a twentieth image feature;

[0416] Inputting the twentieth image feature into the sixteenth network structure for convolution processing and feature enhancement processing to obtain the twenty-first image feature, wherein the sixteenth network structure has the same network structure as the fourteenth network structure;

[0417] Inputting the twenty-first image feature into the seventeenth network structure for convolution processing, feature enhancement processing and down-sampling processing to obtain the twenty-second image feature, wherein the seventeenth network structure is the same as the fifteenth network structure;

[0418] The twenty-second image feature and the eleventh image feature are fused through a fourth feature fusion layer to obtain a twenty-third image feature;

[0419] Inputting the twenty-third image feature into the eighteenth network structure for convolution processing and feature enhancement processing to obtain the twenty-fourth image feature, wherein the eighteenth network structure has the same network structure as the sixteenth network structure;

[0420] The twenty-fourth image feature, the twenty-first image feature and the eighteenth image feature are respectively input into the network head modules corresponding to their respective sizes for classification prediction, so as to obtain the shoulder key points and crotch key points of the target person.

[0421] The device of the embodiment of the present application can be used to detect the key points of the collected first image to be processed, determine the shoulder key points and crotch key points corresponding to the target person, calculate the shoulder-span image distance according to the coordinates of the shoulder key points and crotch key points in the image, and convert the shoulder-span image distance into a first distance representing the actual distance between the target person and the electronic device, so that the target wind speed can be determined according to the first distance, so that the electronic device can adjust the wind speed according to the distance between the user and the electronic device. Compared with the operation method in the related art, the device adjustment method of the embodiment of the present application can be more convenient without the user performing key operations. And the position information and / or gesture information of the target person can be calculated according to the target frame, gesture frame and key points, and then the electronic device can be adjusted according to the position information and / or gesture information of the target person, providing users with a more convenient human-computer interaction method and improving the user experience.

[0422] The present application also provides an electronic device, such as Fig.13 As shown, it includes a processor 1301 , a communication interface 1302 , a memory 1303 and a communication bus 1304 , wherein the processor 1301 , the communication interface 1302 , and the memory 1303 communicate with each other via the communication bus 1304 .

[0423] Memory 1303, used for storing computer programs;

[0424] The processor 1301 is used to implement the following steps when executing the program stored in the memory 1303:

[0425] Acquire a first image to be processed captured by a camera; wherein the first image to be processed includes a target person;

[0426] Performing human body key point detection on the first image to be processed to determine the shoulder key points and crotch key points of the target person in the first image to be processed;

[0427] Calculate the shoulder-span image distance according to the image coordinates of the shoulder key point and the crotch key point in the first image to be processed, and convert the shoulder-span image distance into a first distance, wherein the first distance represents the distance between the target person and the electronic device;

[0428] The preset wind speed correspondence is queried to determine the target wind speed corresponding to the first distance, and the fan is adjusted to the target wind speed, wherein the wind speed correspondence is a preset correspondence between the wind speed and the distance.

[0429] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0430] The communication interface is used for communication between the above electronic device and other devices.

[0431] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0432] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0433] In another embodiment provided in the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned electronic device adjustment methods are implemented.

[0434] In another embodiment provided in the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any electronic device adjustment method in the above embodiments.

[0435] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.

[0436] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0437] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, electronic device, and computer-readable storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0438] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A method for adjusting an electronic device, characterized in that: The electronic device includes a fan and a camera, and the method includes: Acquire a first image to be processed captured by the camera; wherein the first image to be processed includes a target person; Performing human body key point detection on the first image to be processed to determine the shoulder key points and crotch key points of the target person in the first image to be processed; Calculating a shoulder-span image distance according to image coordinates of a shoulder key point and a crotch key point in the first image to be processed, and converting the shoulder-span image distance into a first distance, wherein the first distance represents a distance between the target person and the electronic device; A preset wind speed correspondence relationship is queried to determine a target wind speed corresponding to the first distance, and the fan is adjusted to the target wind speed, wherein the wind speed correspondence relationship is a preset correspondence relationship between wind speed and distance.

2. The method according to claim 1, characterized in that calculating the shoulder-span image distance according to the image coordinates of the shoulder key points and the crotch key points in the first image to be processed, and converting the shoulder-span image distance into the first distance; include: The coordinates of the shoulder key points and / or crotch key points are calculated by the following formula: ； ； Among them, the k x represents the horizontal coordinate of the shoulder key point or the crotch key point, the k y The ordinate of the shoulder key point or the crotch key point is represented by the horizontal and vertical coordinates of the offset coordinates of the key point predicted by the first neural network model relative to the anchor point. kx and t ky The horizontal and vertical coordinates of the anchor point coordinates are c x and c y ; The strides represent the downsampling scaling ratio; The coordinates of the center points of the two shoulder key points and the center points of the two span key points are calculated using the following formula: ； Among them, the k csx represents the horizontal coordinate of the center point of the two shoulder key points, the k lsx represents the horizontal coordinate of the left shoulder key point of the target person, and the k rsx The horizontal coordinate of the key point of the right shoulder of the target person; ； Among them, the k csy represents the ordinate of the center point of the two shoulder key points, the k lsy represents the ordinate of the left shoulder key point of the target person, and the k rsy The vertical coordinate of the key point of the right shoulder of the target person; ； Among them, the k chx represents the horizontal coordinate of the center point of the two cross-part key points, the k lhx represents the horizontal coordinate of the left span key point of the target person, and the k rhx The horizontal coordinate of the right span key point of the target person; ； Among them, the k chy represents the ordinate of the center point of the two cross-part key points, the k lhy represents the ordinate of the left span key point of the target person, and the k rhy The ordinate of the right-span key point corresponding to the target; According to the coordinates of the center points of the two shoulder key points and the coordinates of the center points of the two span key points, the shoulder-span image distance is calculated by the following formula; ； Among them, the d pix represents the shoulder-span image distance; The shoulder-span image distance is converted into the first distance by the following formula; ； Wherein, D represents the first distance, f represents the focal length of the camera, and d body Indicates the preset average shoulder span distance.

3. The method according to claim 1, characterized in that The step of performing human body key point detection on the first image to be processed to determine the shoulder key points and crotch key points of the target person in the first image to be processed includes: Inputting the first image to be processed into a pre-trained first neural network model to obtain shoulder key points and crotch key points of the target person in the first image to be processed; Wherein, the first neural network model further outputs a gesture frame of the target person, and the method further includes: According to the gesture frame, the first image to be processed is segmented to obtain a gesture image; Inputting the gesture image into a pre-trained second neural network model to obtain a gesture classification result corresponding to the gesture image, wherein the gesture classification result indicates a business gesture category corresponding to the gesture in the gesture image; According to the gesture classification result, a device control instruction is determined to enable the fan to perform an adjustment operation corresponding to the device control instruction.

4. The method according to claim 3, characterized in that Before inputting the gesture image into a pre-trained second neural network model to obtain a gesture classification result corresponding to the gesture image, the method further includes: Inputting the gesture image into a pre-trained third neural network model to obtain a clarity score and a blur score of the gesture image; wherein the third neural network model is used to perform clarity detection on the gesture image; When the blur score is greater than or equal to the clarity score, discarding the gesture image; In the case where the clarity score is greater than the fuzziness score, the step of inputting the gesture image into a pre-trained second neural network model is performed to obtain a gesture classification result corresponding to the gesture image.

5. The method according to claim 1, characterized in that After acquiring the first image to be processed collected by the fan, the method further includes: Acquire a second image to be processed captured by the fan, wherein the second image to be processed is an image captured after the first image to be processed; Inputting the first image to be processed and the second image to be processed into a first neural network model respectively to obtain a first target frame and a second target frame, wherein the first target frame represents the position of the target person in the first image to be processed, and the second target frame represents the position of the target person in the second image to be processed; Calculating a third target frame of the target person according to the first target frame, wherein the third target frame represents a predicted position of the target person in the second image to be processed; Matching the second target frame with the third target frame in positional relationship to obtain a matching result; When the matching result indicates that the target person in the first image to be processed and the target person in the second image to be processed are the same target person, calculating the difference between the first target frame and the second target frame to obtain position transformation information; According to the position change information, a device adjustment instruction is determined so that the fan is adjusted according to the device adjustment instruction.

6. The method according to claim 5, characterized in that The step of calculating a third target frame of the target person according to the first target frame includes: Calculating an initial state vector of the target person in the first image to be processed according to the first target frame; Calculating, based on the initial state vector, an error between the first predicted state vector and the first predicted state vector; Calculating the error between the measurement value of the target person in the second to-be-detected image and the measurement value according to the measurement matrix of the measuring device; Calculating a Kalman gain by combining an error in the measurement value with an error in the first predicted state vector; Correcting the first predicted state vector by using the Kalman gain to obtain a first corrected state vector; The third target frame is calculated according to the first corrected state vector.

7. The method according to claim 5, characterized in that The step of matching the second target frame with the third target frame in positional relationship to obtain a matching result comprises: Calculating an intersection-over-union ratio of the second target frame and the third target frame; When the intersection-over-union ratio is greater than or equal to a preset threshold, it is determined that the matching result indicates that the target person in the first image to be processed and the target person in the second image to be processed are the same target person.

8. The method according to claim 3, characterized in that The step of inputting the first image to be processed into a pre-trained first neural network model to obtain the shoulder key points and crotch key points of the target person in the first image to be processed includes: Inputting the first image to be processed into the first neural network model, wherein the first network structure in the first neural network model performs feature extraction and downsampling processing on the first image to be processed to obtain first image features; Inputting the first image feature into a second network structure for convolution processing, feature enhancement processing and downsampling processing to obtain a second image feature; wherein the second network structure is the same as the first network structure; Inputting the second image feature into the third network structure for feature enhancement processing to obtain a third image feature; Inputting the third image feature into a fourth network structure for convolution processing, feature enhancement processing and downsampling processing to obtain a fourth image feature; wherein the fourth network structure is the same as the second network structure; Inputting the fourth image feature into a fifth network structure for feature enhancement processing to obtain a fifth image feature; wherein the fifth network structure is the same as the third network structure; Inputting the fifth image feature into a sixth network structure for convolution processing, feature enhancement processing and downsampling processing to obtain a sixth image feature, wherein the sixth network structure is the same as the fourth network structure; Inputting the sixth image feature into a seventh network structure for feature enhancement processing to obtain a seventh image feature, wherein the seventh network structure is the same as the fifth network structure; Inputting the seventh image feature into an eighth network structure for convolution processing, feature enhancement processing and downsampling processing to obtain an eighth image feature; wherein the eighth network structure is the same as the sixth network structure; Inputting the eighth image feature into a ninth network structure to perform a multi-scale pooling operation, fusing image features of different scales, and obtaining a ninth image feature; Inputting the ninth image feature into a tenth network structure for convolution processing and feature enhancement processing to obtain a tenth image feature, wherein the tenth image feature, the seventh image feature, and the fifth image feature have different sizes; Performing convolution processing and feature enhancement processing on the tenth image feature through an eleventh network structure to obtain an eleventh image feature, wherein the eleventh network structure is the same as the network structure of the eighth network structure; Performing upsampling processing on the eleventh image feature through a first upsampling module to obtain a twelfth image feature, wherein the twelfth image feature has the same size as the seventh image feature; The twelfth image feature and the seventh image feature are subjected to feature fusion through a first feature fusion layer to obtain a thirteenth image feature; Inputting the thirteenth image feature into a twelfth network structure for convolution processing and feature enhancement processing to obtain a fourteenth image feature, wherein the twelfth network structure has the same network structure as the tenth network structure; Inputting the fourteenth image feature into the thirteenth network structure for convolution processing and feature enhancement processing to obtain a fifteenth image feature, wherein the thirteenth network structure is the same as the eleventh network structure; performing upsampling processing on the fifteenth image feature through a second upsampling module to obtain a sixteenth image feature, wherein the sixteenth image feature has the same size as the fifth image feature; The fifth image feature and the sixteenth image feature are subjected to feature fusion by a second feature fusion layer to obtain a seventeenth image feature; Inputting the seventeenth image feature into a fourteenth network structure for convolution processing and feature enhancement processing to obtain an eighteenth image feature; wherein the fourteenth network structure has the same network structure as the twelfth network structure; Inputting the eighteenth image feature into a fifteenth network structure for convolution processing, feature enhancement processing and downsampling processing to obtain a nineteenth image feature, wherein the fifteenth network structure is the same as the thirteenth network structure; The nineteenth image feature is fused with the fifteenth image feature through a third feature fusion layer to obtain a twentieth image feature; Inputting the 20th image feature into a 16th network structure for convolution processing and feature enhancement processing to obtain a 21st image feature, wherein the 16th network structure is the same as the 14th network structure; Inputting the twenty-first image feature into a seventeenth network structure for convolution processing, feature enhancement processing, and downsampling processing to obtain a twenty-second image feature, wherein the seventeenth network structure is the same as the fifteenth network structure; The twenty-second image feature and the eleventh image feature are fused through a fourth feature fusion layer to obtain a twenty-third image feature; Inputting the twenty-third image feature into an eighteenth network structure for convolution processing and feature enhancement processing to obtain a twenty-fourth image feature, wherein the eighteenth network structure is the same as the sixteenth network structure; The twenty-fourth image feature, the twenty-first image feature and the eighteenth image feature are respectively input into the network head modules corresponding to their respective sizes for classification prediction to obtain the shoulder key points and crotch key points of the target person.

9. An electronic equipment adjustment device, characterized in that: The electronic device includes a fan and a camera, and the device includes: A first image to be processed acquisition module, used to acquire a first image to be processed captured by the camera; wherein the first image to be processed includes a target person; A key point detection module, used to perform human key point detection on the first image to be processed, and determine the shoulder key points and crotch key points of the target person in the first image to be processed; A first distance calculation module, configured to calculate a shoulder-span image distance according to image coordinates of shoulder key points and crotch key points in the first image to be processed, and convert the shoulder-span image distance into a first distance, wherein the first distance represents a distance between the target person and the electronic device; The target wind speed determination module is used to query a preset wind speed correspondence, determine the target wind speed corresponding to the first distance, and adjust the fan to the target wind speed, wherein the wind speed correspondence is a preset correspondence between wind speed and distance.

10. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 8 when executing a program stored in a memory.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 8 are implemented.

Images