Electronic device control method, device and electronic device

Abstract

The embodiments of the present specification relate to a control method, device, electronic device and system for an electronic device, the method comprising: acquiring a first image captured by the camera device; detecting a current responding individual in the first image; when the current responding individual is detected, obtaining a gesture state of the current responding individual according to the first image; and obtaining a control instruction issued by the current responding individual to the electronic device as a valid control instruction according to the gesture state.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of interactive technology, and more specifically, to a control method of an electronic device, a control device of an electronic device, an electronic device, and a computer-readable storage medium. Background Art

[0002] With the vigorous development of the interactive technology field in recent years, the mode of human-computer interaction has gradually evolved from contact-based mouse, button and other operations to reduce contact, increase interaction convenience and interaction efficiency. The currently more popular control method is to perform contactless control of electronic devices through set gestures specified by electronic devices. For this kind of contactless control, in the prior art, electronic devices respond accordingly based on the recognition of hand gesture states, without distinguishing the individuals who perform gestures. This will cause response confusion in scenarios where multiple people are operating at the same time, and it will be impossible to respond to the control of a specific individual. Therefore, it is necessary to provide a new technical solution for electronic device control to improve the accuracy of electronic devices in responding to gesture control. Summary of the invention

[0003] One purpose of the embodiments of the present disclosure is to provide a new technical solution for controlling electronic devices.

[0004] According to a first aspect of the present disclosure, an embodiment of a control method of an electronic device is provided. The electronic device has a camera device. The control method includes:

[0005] Acquire a first image captured by the camera device;

[0006] Detecting a specific current responding individual in the first image;

[0007] When the current responding individual is detected, obtaining a gesture state of the current responding individual according to the first image;

[0008] According to the gesture state, the manipulation instruction sent by the current responding individual to the electronic device is obtained as a valid manipulation instruction.

[0009] Optionally, before detecting a specific current responding individual in the image, the method further comprises:

[0010] Acquire a second image captured by the camera device, wherein the second image is captured earlier than the first image;

[0011] detecting an authorized individual in the second image;

[0012] In the case where an authorized individual is detected, an individual with the highest manipulation priority is selected from the authorized individuals as the current responding individual.

[0013] Optionally, detecting the authorized individual in the second image comprises:

[0014] Detecting individuals in the second image, and obtaining a human body region image corresponding to each individual;

[0015] Obtaining a facial region image of a corresponding individual according to the human region image;

[0016] Based on the facial region image, it is detected whether the corresponding individual is an authorized individual.

[0017] Optionally, the detecting individuals in the second image to obtain a human body region image corresponding to each individual includes:

[0018] Inputting the second image into a preset human figure detection model to obtain a human body region corresponding to each individual in the second image;

[0019] According to the human body region, obtaining a corresponding human body region image; and / or,

[0020] The step of obtaining a facial region image of a corresponding individual according to the human region image comprises:

[0021] Inputting the human body region image into a preset face detection model to obtain a face region in the human body region image;

[0022] According to the facial region, a corresponding facial region image is obtained.

[0023] Optionally, before the step of detecting a specific current responding individual in the image, the method further comprises:

[0024] Acquire the picked-up sound signal;

[0025] Detecting the presence of an authorized individual based on the sound signal;

[0026] In the case where an authorized individual is detected, an individual with the highest manipulation priority is selected from the detected authorized individuals as the current responding individual.

[0027] Optionally, the step of determining the current manipulation priority of the authorized individual comprises:

[0028] According to at least one event preset to trigger a priority change, detecting whether one of the events occurs;

[0029] In case an event is detected, obtaining a priority setting scheme corresponding to the event;

[0030] According to the acquired priority setting scheme, the current control priority of the authorized individual is determined.

[0031] Optionally, the at least one event triggering the priority change includes at least one of an event in which the electronic device is in a specified application scenario, an event in which the electronic device receives specific voice content, and an event in which the electronic device senses a specific action.

[0032] Optionally, obtaining, according to the gesture state, a control instruction issued by the current responding individual to the electronic device includes:

[0033] In the case that the gesture state belongs to a set gesture, a manipulation instruction corresponding to the set gesture is acquired as the manipulation instruction issued by the current responding individual to the electronic device.

[0034] Optionally, obtaining the gesture state of the current responding individual according to the first image includes:

[0035] Obtaining a human body region image corresponding to the current responding individual according to the human body region corresponding to the current responding individual in the first image;

[0036] The gesture state of the current responding individual is obtained according to the human body region image.

[0037] Optionally, obtaining the gesture state of the current responding individual according to the human body region image includes:

[0038] According to the human body region image, obtaining a partial image corresponding to the upper body of the current responding individual;

[0039] The gesture state of the current responding individual is obtained according to the partial image.

[0040] Optionally, obtaining the gesture state of the current responding individual according to the partial image includes:

[0041] Extracting a human hand region corresponding to a human hand feature in the partial image to obtain a human hand region image;

[0042] The gesture state of the current responding individual is determined according to the human hand area image.

[0043] Optionally, determining the gesture state of the current responding individual according to the human hand region image includes:

[0044] Inputting the human hand region image into a preset classification model, wherein the classification model reflects a mapping relationship between the human hand image and a classification result about the gesture;

[0045] According to the classification result output by the classification model, the gesture state of the current responding individual is obtained.

[0046] Optionally, the step of obtaining the classification model includes:

[0047] Acquire a human hand image sample, wherein the human hand image sample has a label related to a corresponding gesture state;

[0048] Training the model parameters of the selected model using the human hand image samples;

[0049] The selected model is configured according to the model parameters obtained through training to obtain the classification model.

[0050] Optionally, detecting a specific current responding individual in the first image includes:

[0051] Acquire optical flow information between the first image and a previous image, wherein the previous image is an image captured earlier than the first image and in which the current responding individual is detected;

[0052] The current responding individual is detected in the first image according to the position coordinates of the current responding individual in the previous image and the optical flow information.

[0053] Optionally, detecting a specific current responding individual in the first image includes:

[0054] Extracting pixel features of the current responding individual in a preceding image captured by the camera device, wherein the preceding image is an image captured earlier than the first image and in which the current responding individual is detected;

[0055] The current responding individual is detected in the first image according to the pixel features.

[0056] According to a second aspect of the present specification, an embodiment of a control method of another electronic device is also provided. The electronic device includes a camera device. The control method includes:

[0057] Tracking and detecting the gesture state of a specific current responding individual in a sequence of images captured by the camera device, wherein the current responding individual is an individual with the highest control priority in the first frame of the image sequence;

[0058] According to the gesture state, obtaining a control instruction issued by the current responding individual to the electronic device;

[0059] According to the control instruction, the electronic device is controlled to respond.

[0060] Optionally, before tracking and detecting a gesture state of a specific current responding individual in a sequence of images captured by the camera device, the method further includes:

[0061] detecting an authorized individual in an image captured by the camera device;

[0062] In the case where an authorized individual is detected, an individual with the highest manipulation priority is selected from the authorized individuals as the current responding individual, and the image is used as the first frame image.

[0063] Optionally, the method further comprises:

[0064] In the case that the current responding individual is not detected in the subsequent image, taking the subsequent image as the first image, and executing the step of detecting the authorized individual in the first image captured by the camera device;

[0065] The shooting time of the subsequent image is later than the shooting time of the first frame image in the image sequence.

[0066] According to a third aspect of the present specification, an embodiment of a control device for an electronic device is also provided, including:

[0067] An image acquisition module, used to acquire a first image captured by the camera device;

[0068] A tracking and detection module, used for detecting a specific current responding individual in the first image;

[0069] a gesture recognition module, configured to obtain a gesture state of the current responding individual according to the first image when the current responding individual is detected; and

[0070] The control execution module is used to obtain the control instruction sent by the current responding individual to the electronic device according to the gesture state as a valid control instruction.

[0071] According to a fourth aspect of the present specification, an embodiment of an electronic device is further provided, including the control device of the electronic device as described in the third aspect of the present specification, or the electronic device includes:

[0072] A memory for storing executable commands;

[0073] The processor is used to execute the control method of the electronic device as described in the first aspect or the second aspect of this specification under the control of the executable command.

[0074] Wherein, the camera device is connected to the processor to output the captured image to the processor.

[0075] Optionally, the electronic device further includes a display device, and a screen of the display device is greater than or equal to 50 inches.

[0076] According to the fifth aspect of the present specification, an embodiment of a computer-readable storage medium is also provided, which stores executable instructions. When the executable instructions are executed by a processor, the control method of the electronic device as described in the first aspect or the second aspect of the present specification is executed.

[0077] In one embodiment of the present disclosure, before the electronic device recognizes the gesture state according to the image captured by the camera device, it will determine a specific individual as the current responding individual, and track and detect the current responding individual in the image captured by the camera device to obtain the gesture state of the current responding individual, and use the control instruction corresponding to the gesture state as a valid control instruction to control the electronic device to respond according to the control instruction. This can effectively avoid the response confusion problem that may be caused by multiple people operating at the same time.

[0078] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0080] Figure 1 A schematic diagram of an application scenario according to an embodiment is illustrated;

[0081] Figure 2 is a hardware configuration diagram of an electronic device that can be used to implement a control method of an embodiment;

[0082] Figure 3 is a flowchart of a control method of an electronic device according to an embodiment;

[0083] Figure 4 is a flow chart of a control method of an electronic device according to another embodiment;

[0084] Figure 5 is a flow chart of a control method of an electronic device according to an example;

[0085] Figure 6 is a principle block diagram of a control device of an electronic device according to an embodiment;

[0086] Figure 7is a schematic structural diagram of an electronic device according to an embodiment. DETAILED DESCRIPTION

[0087] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention unless otherwise specifically stated.

[0088] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0089] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0090] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0091] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0092] In an application scenario where a user controls an electronic device through a set gesture specified by the electronic device, the electronic device can capture an image in front of it through a camera device, and obtain the user's gesture state based on the captured image. When the obtained gesture state belongs to a set gesture, the electronic device is controlled to respond according to the control instruction corresponding to the gesture state.

[0093] In one embodiment, the electronic device can directly detect the existing hand features in the image captured by the camera device to obtain the gesture state corresponding to the hand features. In this case, when there are multiple hand features in the image and the gesture states corresponding to the multiple hand features all belong to the set gesture states, the electronic device may have a problem of confused response.

[0094] In order to solve this problem, the embodiment of the present disclosure proposes a technical solution for tracking a determined current responding individual and controlling an electronic device to respond according to the gesture state of the current responding individual, thereby avoiding the problem of response confusion in the electronic device. At the same time, the effect of controlling the electronic device to respond based on the control requirements of a specific current responding individual is achieved, so as to improve the targeting and accuracy of the response.

[0095] Figure 1An application scenario of an embodiment of the present disclosure is shown. In this application scenario, the electronic device 1000 tracks the current responding individual in the video stream captured by the camera device 1500, so as to control the electronic device 1000 to respond according to the gesture state of the current responding individual, wherein the video stream can be understood as a sequence of images arranged in chronological order of shooting time. The electronic device 1000 can be any device having a camera device 1500, for example, a mobile phone, a television, a projector, or a speaker, etc., which is not limited here.

[0096] After the electronic device 1000 is turned on, the camera device 1500 is started to capture images at a set shooting frequency, and the current responding individual and the gesture state of the current responding individual are detected based on the images captured by the camera device 1500. When the electronic device 1000 starts the camera device 1500 to capture, the camera device 1500 continuously captures multiple frames of images at a set frequency, thereby obtaining the image sequence. The electronic device 1000 can determine the current responding individual based on the changes in the individuals in the image sequence, and track the current responding individual to continuously respond to the current responding individual.

[0097] by Figure 1 For example, the i-th frame image of the image sequence shows individual 1 and individual 2, wherein both individual 1 and individual 2 are authorized individuals, and the control priority of individual 1 is higher than that of individual 2. The electronic device 1000 can determine that individual 1 is the current responding individual who currently needs to respond based on the i-th frame image, and the electronic device 1000 will continue to track the individual 1 in subsequent images, for example, the individual 1 can be tracked in the subsequent (i+1)-th frame and (i+2)-th frame image, until the individual 1 is not tracked according to the j-th frame image, and the response stage of using individual 1 as the current responding individual is ended, wherein j is an integer greater than (i+2). In this response stage, the electronic device 1000 will control the electronic device 1000 to respond according to the gesture state of the individual 1. Based on the j-th frame image, the electronic device 1000 can re-determine individual 2 as the current responding individual who currently needs to respond, and continue to track individual 2 in subsequent images to control the electronic device 1000 to respond according to the gesture state of individual 2, until individual 2 cannot be tracked in a subsequent frame image, ending the response stage of taking individual 2 as the current responding individual, and so on.

[0098] It can be seen that through the control method of the embodiment of the present disclosure, not only can a specific individual be selected for tracking response, but also the problem of response confusion can be avoided when multiple people operate at the same time.

[0099] <Hardware Configuration>

[0100] Figure 2FIG. 1 is a schematic diagram of the structure of an electronic device 1000 to which the control method according to an embodiment of the present disclosure can be applied.

[0101] like Figure 2 As shown, the electronic device 1000 may include a processor 1100, a memory 1200, an interface device 1300, a communication device 1400, a camera device 1500, a display device 1600, and the like. The processor 1100 is used to execute a computer program, which may use an instruction set of an architecture such as x86, Arm, RISC, MIPS, SSE, etc. The memory 1200 includes, for example, a ROM (read-only memory), a RAM (random access memory), a non-volatile memory such as a hard disk, and the like. The interface device 1300 includes, for example, a USB interface, a headphone interface, and the like. The communication device 1400 is capable of, for example, wired or wireless communication, long-distance or short-distance communication. The camera device 1500 includes at least one camera for capturing images. The display device 1600 is used to display the content to be played.

[0102] The electronic device 1000 may be a terminal device such as a smart speaker, a smart TV, a projector, etc. The electronic device 1000 may be a terminal device with a larger display screen, for example, the screen of the display device 1600 of the electronic device 1000 is greater than or equal to 50 inches, etc., which is not limited here.

[0103] In this embodiment, the memory 1200 of the electronic device 1000 is used to store a computer program, and the computer program is used to control the processor 1100 to operate to implement the control method according to any embodiment. A technician can design a computer program according to the scheme disclosed in this specification. How the computer program controls the processor 1100 to operate is well known in the art, so it will not be described in detail here.

[0104] It should be understood by those skilled in the art that although Figure 2 Multiple devices of the electronic device 1000 are shown in the figure, but the electronic device 1000 of the embodiment of the present disclosure may only involve some of the devices, or may also include other devices, which is not limited here.

[0105] <Method Example 1>

[0106] Figure 3 A control method of an electronic device according to an embodiment is shown. The control method may be performed as follows. Figure 2 The electronic device 1000 is shown implemented.

[0107] The electronic device 1000 of this embodiment has a camera device 1500. Figure 3 As shown, the control method of the electronic device 1000 may include the following steps S3100 to S3400:

[0108] Step S3100, acquiring a first image captured by the camera device 1500.

[0109] In step S3100, the first image may be a frame of image captured by the camera device 1500 after the electronic device 1000 is turned on.

[0110] Step S3200, detecting a current responding individual in the first image.

[0111] In this step S3200, the current responding individual is the individual currently responding by the electronic device.

[0112] In this embodiment, the electronic device 1000 has determined the current responding individual before executing step S3200.

[0113] The current responding individual may be specified by the user during the current use of the electronic device 1000 , wherein the user may change the specified current responding individual during the current use of the electronic device.

[0114] For example, before executing step S3200, the electronic device may further execute the following steps: according to the operation of selecting the current responding individual, start the camera device to capture and save the image of the selected individual so that the electronic device can determine the current responding individual.

[0115] The current responding individual may also be determined by the electronic device based on stored authorization information. For example, the current responding individual may be determined based on a facial image of an authorized individual in the authorization information and a corresponding control priority, etc. This is not limited here.

[0116] In one embodiment, the electronic device 1000 can determine the current responding individual based on the second image captured before the first image in combination with the authorization information, and track and detect the determined current responding individual in the first image, wherein the first image is, for example, Figure 1 The (i+1)th frame image or the (i+2)th frame image in the image, etc., the second image is, for example, Figure 1 The i-th frame image in .

[0117] In one embodiment, the step S3200 of detecting a specific current responding individual in the first image may include the following steps S3211 to S3212:

[0118] Step S3211, obtaining optical flow information between the first image and the previous image, wherein the previous image is taken earlier than the first image and is detected

[0119] In step S3211, the preceding image may be a frame image preceding the first image, or may be another frame image in a sequence of images of the current responding individual, such as the first frame image in the sequence of images, wherein the first image belongs to the sequence of images. Figure 1 The (i+1)th frame image in the image, the previous image is Figure 1 For example, the first image is Figure 1 The (i+2)th frame image in the image, the previous image is Figure 1 The optical flow is the instantaneous speed of the pixel movement of the spatial moving object on the observation imaging plane. Here, the optical flow information between the first image and the previous image can be obtained according to any optical flow algorithm for acquisition in step S3211.

[0120] The optical flow algorithm is a method that uses the change of pixels in the image sequence in the time domain and the correlation between adjacent frames to find the corresponding relationship between the previous frame and the current frame, thereby calculating the motion information of the target object between adjacent frames. In step S3211, the target object is also the current responding individual.

[0121] Step S3212: Detect the current responding individual in the first image according to the position coordinates and optical flow information of the current responding individual in the previous image.

[0122] According to the above steps S3211 to S3212, in this embodiment, since the optical flow information reflects the motion information of the current responding individual between the first image and the previous image, the positional relationship between the image position coordinates of the current responding individual in the first image and the image position coordinates in the previous image can be obtained based on the optical flow information. In this way, based on the optical flow information and the image position coordinates of the current responding individual in the previous image, the image position coordinates of the current responding individual in the first image can be obtained, thereby achieving the purpose of detecting the current responding individual in the first image.

[0123] According to the above steps S3211 to S3212, it can be known that by tracking and detecting the current responding individual in the first image through optical flow information, there is no need to perform human shape detection and face detection on the first image through a human shape detection model, etc., so that the current responding individual can be detected through facial image comparison, which will increase the detection speed and further increase the response speed to gesture control.

[0124] In one embodiment, detecting a specific current responding individual in the first image in step S3200 may also include the following steps S3021-S3022:

[0125] Step S3021, extracting pixel features of the current responding individual in a previous image captured by a camera device, wherein the previous image is an image captured earlier than the first image and in which the current responding individual is detected.

[0126] The preceding image may be a frame image preceding the first image, or may be another frame image in an image sequence of the current responding individual, such as the first image in the image sequence, etc., wherein the first image belongs to the image sequence.

[0127] The pixel features extracted in step S3021 may include, for example, contour features of the current responding individual, etc., which are not limited here.

[0128] Step S3022: Detect the current responding individual in the first image according to the extracted pixel features.

[0129] In step S3022, whether the first image contains the current responding individual can be determined by detecting whether the extracted pixel features exist in the first image, so as to improve the detection speed.

[0130] In another embodiment, the above optical flow information and pixel features may also be combined to detect the current responding individual in the first image, which is not limited here.

[0131] Step S3300: when a current responding individual is detected, the gesture state of the current responding individual is obtained according to the first image.

[0132] In this embodiment, the electronic device will track the current responding individual in the first image when the current responding individual has been determined, and obtain the gesture state of the current responding individual according to the first image when the current responding individual is tracked, so as to obtain the control requirements of the current responding individual according to the gesture state. When obtaining the gesture state of the current responding individual according to the first image, if the distance between the current responding individual and the camera device 1500 is far, for example, the distance between the current responding individual and the camera device 1500 is 3 to 5 meters, the current responding individual will have a smaller proportion in the first image, and the hand of the current responding individual will have an even smaller proportion in the first image, which will increase the difficulty of accurately identifying the gesture state of the current responding individual according to the first image and reduce the accuracy of the recognition result. In order to solve the problem of long-distance recognition, in one embodiment, obtaining the gesture state of the current responding individual according to the first image in step S3300 may include the following steps S3310 to S3320:

[0133] Step S3310, obtaining a human body region image corresponding to the current responding individual based on the human body region corresponding to the current responding individual in the first image.

[0134] In this embodiment, the human body region of the first image corresponding to the current responding individual may be a square region where the current responding individual is located, and the square region may be a rectangle or a square. The human body region may be obtained in the above step S3200 of "detecting a specific current responding individual in the first image", that is, when the current responding individual is detected in the first image by step S3200, the position region of the current responding individual in the first image may be obtained at the same time, that is, the human body region of the first image corresponding to the current responding individual may be obtained.

[0135] Step S3320: obtaining the gesture state of the current responding individual according to the human body area image.

[0136] In step S3320, since the current responding individual occupies a large proportion in the human body area image, obtaining the gesture state of the current responding individual based on the human body area image will be able to improve the accuracy of the recognition result of the gesture state.

[0137] In one embodiment, in order to further improve the accuracy of the recognition result of the gesture state of the current responding individual, obtaining the gesture state of the current responding individual according to the human body region image in step S3320 may include the following steps S3321-S3322:

[0138] Step S3321, obtaining a partial image of the upper body corresponding to the current responding individual based on the human body region image.

[0139] In step S3321, the area where the upper body of the current responding individual is located can be determined in the human body area image based on the relationship between the human body coordinate position and the face coordinate position in the human body area image. The area is, for example, a square area, and the upper body should include the hands of the current responding individual.

[0140] In step S3321, the partial image can be obtained by cutting out the area where the upper body of the current responding individual is located in the human body area image.

[0141] Step S3322: Obtain the gesture state of the current responding individual according to the local image.

[0142] In step S3322, since the hands of the current responding individual will occupy a larger proportion in the local image compared to the human body area image, obtaining the gesture state of the current responding individual based on the local image will help improve the accuracy of the recognition result.

[0143] In one embodiment, obtaining the gesture state of the current responding individual based on the local image in step S3322 may include: inputting the local image into a preset classification model, through which the classification model can detect the area where the hand is located in the local image, and outputting the corresponding gesture state based on the hand features reflected in the area.

[0144] In one embodiment, in order to further improve the accuracy of the recognition result of the gesture state of the current responding individual, the step S3322 of obtaining the gesture state of the current responding individual according to the partial image may include the following steps S3322-1 to S3322-2:

[0145] Step S3322-1, extract the hand area corresponding to the hand features in the local image to obtain the hand area image.

[0146] In one embodiment, the local image can be input into a preset hand detection model, which can detect a hand area corresponding to hand features in the local image, and then obtain an image of the hand area by cutting out the hand area in the local image.

[0147] In this embodiment, the step of obtaining the human hand detection model may include: obtaining positive samples and negative samples about human hand images; training model parameters of a selected model based on these samples; and configuring the selected model based on the model parameters obtained by training, that is, obtaining the human hand detection model.

[0148] The selected model may be a model of any structure suitable for image detection, for example, a neural network model, etc., which is not limited here.

[0149] Step S3322-2, determine the gesture state of the current responding individual based on the human hand area image.

[0150] In step S3322-2, since the hand of the current responding individual will occupy a larger proportion in the hand area image compared to the local image, obtaining the gesture state of the current responding individual based on the hand area image will help improve the accuracy of the recognition result.

[0151] In step S3322-2, the gesture state of the current responding individual is determined based on the hand area image, which may include the following steps: inputting the hand area image into a preset classification model, wherein the classification model reflects the mapping relationship between the hand image and the classification result of the gesture; and, based on the classification result output by the classification model, obtaining the gesture state of the current responding individual.

[0152] The classification model can be obtained by pre-training, wherein the steps of obtaining the classification model may include: obtaining a human hand image sample, wherein the human hand image sample has a label regarding the corresponding gesture state; training the model parameters of the selected model through the human hand image sample; and configuring the selected model according to the trained model parameters to obtain the classification model.

[0153] In this embodiment, the selected model may be a model of any structure suitable for image classification, for example, a multi-classification model of any structure, etc., which is not limited here.

[0154] In this embodiment, different models can be obtained by training based on different selected models. For example, the hand detection model and the classification model are respectively obtained by training based on different selected models.

[0155] Step S3400, based on the gesture state obtained in step S3300, obtain the control instruction currently issued by the responding individual to the electronic device 1000 as a valid control instruction.

[0156] In this embodiment, the electronic device 1000 recognizes the gesture state and converts the gesture state into a corresponding control instruction to execute the control instruction.

[0157] In this embodiment, the electronic device 1000 regards the operation instruction issued by the current responding individual as a valid control instruction, and then executes the control instruction.

[0158] In one embodiment, in step S3400, obtaining the control instruction issued by the current responding individual to the electronic device 1000 according to the gesture state may include: when the gesture state belongs to a set gesture, obtaining the control instruction corresponding to the set gesture as the control instruction issued by the current responding individual to the electronic device 1000.

[0159] In this embodiment, the control instructions corresponding to the set gestures can be set before the electronic device 1000 leaves the factory, and the user can control the electronic device according to the set gestures corresponding to the control instructions provided in the manual provided with the electronic device 1000.

[0160] In this embodiment, the electronic device may also support the user to customize the set gestures corresponding to each control instruction according to his / her own operating habits.

[0161] When the gesture state does not belong to the set gesture, the processing of the current first image is terminated, and the process returns to step S3100 to obtain a new first image, and further executes the above steps S3200 to S3400 to achieve tracking response to the gesture state of the current responding individual.

[0162] In this embodiment, an image captured by the camera device 1500 is used as an example of the first image to illustrate the control method. When the electronic device 1000 uses other images as the first image to implement the control method, it can refer to the above steps S3100 to S3400, which will not be repeated here.

[0163] In this embodiment, if the current responding individual is not detected in the first image, the tracking of the current responding individual will be terminated, and a new current responding individual can be determined based on the first image. In the case that there is no new current responding individual in the first image, the new current responding individual will be locked according to the subsequent image captured by the camera device 1500, and after locking the new current responding individual, the gesture state of the new current responding individual will be tracked and responded to according to steps S3100 to S3400. According to the above steps S3100 to S3400, it can be known that when the electronic device 1000 performs gesture recognition based on the first image, it will predetermine the current responding individual, that is, determine the current responding individual, and track and detect the current responding individual in the first image to obtain the gesture state of the current responding individual, and use the control instruction corresponding to the gesture state as a valid control instruction to control the electronic device to respond according to the control instruction. This can effectively avoid the response confusion problem that may be caused by multiple people operating at the same time.

[0164] In one embodiment, the method may further include the following steps S3011 to S3013 before the above step S3200 of detecting a specific current responding individual in the first image:

[0165] Step S3011, acquiring a second image captured by the camera device 1500, wherein the shooting time of the second image is earlier than the shooting time of the first image.

[0166] The second image may be a frame of image that needs to determine the current responding individual. The second image may be an image used by the electronic device to determine the current responding individual for the first time after the electronic device is turned on, or an image used to re-determine the current responding individual, which is not limited here. For example, the second image is Figure 1 The i-th frame image in .

[0167] Step S3012: Detecting an authorized individual in the second image.

[0168] The authorized individual is an individual with the authority to control the electronic device. The user can input authorization information to the electronic device through the authorization operation supported by the electronic device, and the authorization information includes the image of the individual allowed to control the electronic device and the control priority of the corresponding individual, so that the electronic device can perform step S3012 of detecting the authorized individual in the second image.

[0169] When detecting the authorized individual in the second image, if the distance between each individual in front of the electronic device 1000 and the camera device 1500 is far, for example, the distance between the individual and the camera device 1500 is 3 to 5 meters, then the individual will have a smaller proportion in the second image, and the face of the individual will have an even smaller proportion in the second image, which will increase the difficulty of detecting the authorized individual based on the second image, thereby reducing the accuracy of the detection result. To solve this problem, in one embodiment, the step S3012 of detecting the authorized individual in the second image may include the following steps S3012a to S3012c:

[0170] Step S3012a, detecting individuals in the second image, and obtaining a human body region image corresponding to each individual.

[0171] In step S3012a, the electronic device 1000 detects all individuals in the second image to obtain a human body region image corresponding to each individual. This can be done by detecting the individuals in the second image, obtaining a rectangular region where the individuals are located, and cutting out the square region on the second image to obtain a human body region image corresponding to the individuals.

[0172] In one embodiment, the step S3012a detects individuals in the second image and obtains a human body region image corresponding to each individual, which may include the following steps S3012-1 to S3012-2:

[0173] Step S3012-1, input the second image into a preset human figure detection model to obtain a human body region corresponding to each individual in the second image.

[0174] The human figure detection model can be pre-trained and stored in the electronic device 1000 for implementation of step S3012-1.

[0175] In this embodiment, the human shape detection model can detect and mark the human body area in the second image. For example, a square frame corresponding to the human body area can be marked on the second image.

[0176] In one embodiment, the step of obtaining the human figure detection model may include: obtaining positive samples and negative samples of human figure images, wherein these samples have labels as to whether they are human figures; training model parameters of a selected model through the human figure image samples; and configuring the selected model according to the model parameters obtained through the training to obtain the human figure detection model.

[0177] In this embodiment, any model structure suitable for human detection can be selected as the selected model. The model structure can be a convolutional neural network structure, such as a ResNet network structure, a VGG network structure, a Densenet network structure, etc., which is not limited here.

[0178] In one embodiment, the positive samples and negative samples of human images can be divided into training samples and test samples according to a set ratio, and the model parameters of the selected model can be trained through the training samples, and the performance of the trained human detection model can be verified through the test samples. If the model performance cannot meet the set requirements, the model parameters of the selected model will be re-trained or continued to be trained through the training samples until the model performance of the trained human detection model meets the set requirements.

[0179] Step S3012-2: Obtain a corresponding human body region image according to the human body region.

[0180] In step S3012-2, the portion where the human body region is located is cut out from the second image to obtain a human body region image. For any human body region image obtained, the proportion of the corresponding individual in the human body region image will be much greater than the proportion in the second image. Therefore, detecting whether the corresponding individual is an authorized individual based on the human body region image will have higher accuracy.

[0181] Step S3012b: obtaining a facial region image of the corresponding individual according to the human region image.

[0182] In step S3012b, obtaining the facial region image of the corresponding individual based on the human region image may include the following steps: inputting the human region image into a preset face detection model to obtain the facial region in the human region image; and, based on the facial region, obtaining the corresponding facial region image.

[0183] In this embodiment, the face detection model can detect and mark the location of the face region in the human body region image, for example, a rectangular frame corresponding to the face region can be marked on the human body region image. In this way, the face region image can be obtained by cutting out the portion corresponding to the face region in the human body region image.

[0184] According to step S3012b, for any facial area image obtained, the proportion of the corresponding individual's face in the facial area image will be much greater than the proportion in the second image and the human body area image. Therefore, detecting whether the corresponding individual is an authorized individual based on the facial area image will have higher accuracy.

[0185] In this embodiment, the step of obtaining the face detection model can be implemented by referring to the above-mentioned step of obtaining the human figure detection model, which will not be described in detail here.

[0186] Step S3012c: Detect whether the corresponding individual is an authorized individual based on the facial area image.

[0187] In this embodiment, for example, the facial image of the authorized individual can be saved in a facial library, and a correspondence between the authorized individual and the facial image can be established. The authorized individual can be one or more. When there are multiple authorized individuals, different control priorities can be set for different authorized individuals.

[0188] The facial image entered by the user may be a facial image uploaded by the user when performing an authorization operation or captured by the camera device 1500 , which is not limited here.

[0189] In step S3012c, for any facial area image obtained, the facial area image can be matched in the face library. When the facial area image is matched, the corresponding individual currently responding individual is determined to be an authorized individual. When the facial area image is not matched, the corresponding individual is determined to be an unauthorized individual.

[0190] Step S3013, when an authorized individual is detected, an individual with the highest control priority is selected from the authorized individuals as the current responding individual.

[0191] In step S3013, when one authorized individual is detected, the authorized individual is the individual with the highest manipulation priority. When at least two authorized individuals are detected, the individual with the highest manipulation priority can be selected as the current responding individual by comparing the current manipulation priorities of the at least two authorized individuals.

[0192] When the electronic device 1000 detects the current responding individual based on the second image, it can refer to the above steps S3300 and S3400 to obtain the gesture state of the current responding individual in the second image, and obtain the control instructions issued by the current responding individual to the electronic device based on the gesture state.

[0193] Furthermore, when the electronic device 1000 detects the current responding individual based on the second image, it will use each frame image taken after the second image as the above first image, and execute the above steps S3100 to S3400 until the current responding individual cannot be detected in an image taken after the second image, thereby ending the tracking response to the current responding individual.

[0194] The method may also determine the current responding individual to be tracked based on the sound signal picked up by the electronic device to perform the above step S3200. In this embodiment, the method may further include the following steps S3021 to S3023 before the above step S3200 of detecting a specific current responding individual in the first image:

[0195] Step S3021, obtaining the picked-up sound signal.

[0196] In this embodiment, the electronic device may be provided with a microphone, and the electronic device may pick up external sound signals through the microphone.

[0197] The sound signal picked up in step S3021 may include a sound signal from one user, or may include sound signals from at least two users.

[0198] Step S3022: Detect the existence of an authorized individual based on the sound signal.

[0199] In this embodiment, when performing the authorization operation, the user can input the authorized voice and the facial image of the authorized individual into the electronic device to form the authorization information, and the authorization information establishes a mapping relationship between the authorized voice and the facial image. When the electronic device receives a confirmation signal for the authorization information, the authorized individual becomes an authorized individual.

[0200] In step S3022, based on the sound features of the acquired sound signal, the individuals involved in the sound signal and the sound features of the individuals involved can be determined. In this way, by matching the sound features of each individual with the sound features corresponding to the authorized voice in the authorization information, it is possible to detect whether there is an authorized individual among the individuals involved.

[0201] Step S3023, when an authorized individual is detected, an individual with the highest control priority is selected from the detected authorized individuals as the current responding individual.

[0202] In step S3023, when one authorized individual is detected, the authorized individual is the individual with the highest manipulation priority. When at least two authorized individuals are detected, the individual with the highest manipulation priority can be selected as the current responding individual by comparing the current manipulation priorities of the at least two authorized individuals.

[0203] In this embodiment, when the individual with the highest control priority is selected through step S3023, the facial image of the individual can be obtained according to the authorization information of the individual, and then the current responding individual can be identified in the first image based on the facial image, and then the tracking response to the current responding individual can be achieved.

[0204] In one embodiment, in one embodiment, the control priority of the authorized individual may be pre-set by the user, and the setting may be to set a fixed control priority for the authorized individual. In this case, if the user needs to adjust the control priority of the authorized individual, manual modification is required.

[0205] In order to improve the convenience of changing the control priority, in one embodiment, the electronic device can also be configured to allow the user to set the event that triggers the priority change and the priority setting scheme corresponding to each event. In this way, the electronic device can adaptively change the current control priority of the authorized individual according to the event and the corresponding priority setting scheme.

[0206] In this embodiment, the step of determining the current control priority of the authorized individual may include the following steps S3031-S3032:

[0207] Step S3031, based on at least one event preset to trigger a priority change, detecting whether one of the events occurs.

[0208] The at least one event may include at least one of an event that the electronic device is in a specified application scenario, an event that the electronic device receives specific voice content, and an event that the electronic device senses a specific action.

[0209] The specified application scenario includes, for example, at least one of a scenario of playing an animation channel, a game scenario, a scenario of playing a movie channel, etc. For example, in the scenario of playing an animation channel, the corresponding priority setting scheme is that children have the highest control priority. For another example, in the game scenario, the corresponding priority setting scheme is that a father or a mother has the highest control priority, etc.

[0210] The specific voice content includes, for example, "voice content with the highest priority regarding the establishment of a specific individual" and the like.

[0211] The specific action may be any pre-set action, as long as it is distinguishable from a gesture used to control an electronic device, and is not limited here.

[0212] Step S3032: when an event is detected, obtain a priority setting scheme corresponding to the event.

[0213] When more than two events are detected, a priority setting scheme corresponding to the highest priority event may be obtained according to the update priorities of these events. For example, a specific voice content or a specific action has the highest update priority among all events.

[0214] Step S3032: Determine the current control priority of the authorized individual according to the acquired priority setting scheme.

[0215] After the current manipulation priority of the authorized individuals is determined, the individual with the highest manipulation priority can be selected as the current responding individual in executing the above step S3013 or S3023.

[0216] <Method Example 2>

[0217] Figure 4 A control method of an electronic device 1000 according to an embodiment is shown. The control method can be implemented by the electronic device, for example, Figure 2 The electronic device 1000 is shown implemented.

[0218] The electronic device 1000 of this embodiment has a camera device 1500. Figure 4 As shown, the control method of the electronic device 1000 may include the following steps S4100 to S4300:

[0219] Step S4100, in a sequence of images captured by the camera device 1500, tracking and detecting the gesture state of a specific current responding individual, wherein the current responding individual is an individual with the highest control priority in the first frame of the image sequence.

[0220] In this embodiment, the electronic device 1000 detects an authorized individual in the first frame image, and uses the authorized individual with the highest control priority as the current responding individual, and tracks and detects the gesture state of the current responding individual in a sequence of images starting with the first frame image to execute the control instruction issued by the current responding individual through gestures. Figure 1 The i-th frame image in .

[0221] The first frame image is the second image, and each frame image after the first frame image in the image sequence is the first image. Each frame image in the image sequence has the current response individual, and the next frame image in the image sequence does not have the current response individual.

[0222] In step S4100, referring to the above step S3300, the gesture state of the current responding individual can be obtained for each frame image of the image sequence in the order of shooting time.

[0223] Step S4200, obtaining the control instruction issued by the current responding individual to the electronic device 1000 according to the gesture state.

[0224] Step S4300: Control the electronic device 1000 to respond according to the control instruction.

[0225] In this embodiment, the electronic device 1000 matches the control instructions in the system according to the gesture state, and if the match is successful, the electronic device 1000 responds.

[0226] In this embodiment, when the matching fails, the processing of the current frame of image will be terminated.

[0227] According to the above steps S4100 to S4300, when the electronic device performs gesture recognition based on the image collected by the shooting device, once based on the second image, for example Figure 1 The current responding individual is determined in the i-th frame image in the image sequence with the i-th frame image as the first frame image, and the current responding individual is tracked and detected to obtain the gesture state of the current responding individual, until the current responding individual is not detected in the j-th frame image, indicating that the image sequence ends at the (j-1)-th frame image. For example, the current responding individual can be detected in 50 consecutive frames after the second image, and the current responding individual cannot be detected in the 51st frame image after the second image, then the length of the image sequence is 51 frames. Further, the electronic device 1000 starts from Figure 1 Starting from the j-th frame image in the image processing unit, a new current responding individual will be determined, and when a new current responding individual is determined based on a certain frame image, steps S4100 to S4300 will be executed starting from the certain frame image, and the cycle will be repeated until the electronic device 1000 is turned off.

[0228] In one embodiment, before tracking and detecting the gesture state of a specific current responding individual in the above image sequence captured by the camera device 1500, the method may further include the following steps S4010 to S4020:

[0229] Step S4010 , detecting an authorized individual in an image captured by the camera device 1500 .

[0230] Step S4020, when an authorized individual is detected, an individual with the highest manipulation priority is selected from the authorized individuals as the current responding individual, and the image is used as the first frame image in step S4100.

[0231] In one embodiment, the method may further include the following steps: when the current responding individual is not detected in the subsequent image, the subsequent image is used as an image in the above step S4010, and step S4010 of detecting the authorized individual in an image captured by the camera device 1500 is executed.

[0232] In this embodiment, the shooting time of the subsequent image is later than the shooting time of the first frame image in the image sequence.

[0233] In this embodiment, when the current responding individual being tracked and detected is lost, step S4010 is re-executed to update the current responding individual and start tracking the updated current responding individual.

[0234] <Example>

[0235] by Figure 2 For example, the electronic device 1000 in FIG. 1 responds to a control instruction issued by a user to a TV through a gesture, and performs a corresponding operation according to the control instruction. The electronic device 1000 has a camera device 1500, which can be set in the middle of the upper edge of the TV, etc., which is not limited here. Figure 5 A flow chart of a control method according to an example implemented by the electronic device 1000 is provided.

[0236] Step S5100, initializing the system.

[0237] The electronic device 1000 may perform initial settings in step S5100. For example, the initial settings may include setting the resolution of the camera device, where the resolution of the camera device may be set to at least 1080*720. In this example, the default resolution may be used, or a plurality of optional resolutions may be provided to the user to determine the resolution of the camera device according to the user's selection. For another example, the initial settings may also include setting the shooting frequency of the camera device shooting images.

[0238] The initial setting may also include authorization registration of individuals who are allowed to operate the electronic device. During the authorization registration, the electronic device 1000 may provide an authorization registration interface according to the user's authorization operation, wherein the authorization registration interface may include an interface for uploading images and an interface for setting the control priority, etc.; the electronic device obtains the authorization information input by the user through the authorization registration interface, wherein the authorization information includes the facial image of the individual who is allowed to operate the electronic device and the control priority of the individual, and records the authorization information to complete the authorization registration, at which point the individual becomes an authorized individual.

[0239] The user can perform the above operations to authorize multiple individuals. For example, a family has 4 people (father, mother, and two children). If the electronic device 1000 is only allowed to respond to the control of "father", the above authorization registration can be performed only for "father". If it is necessary to support multi-person control, the above authorization registration can be performed separately for father and mother.

[0240] In the case where only one individual is allowed to operate the electronic device, the operation priority may not be entered during the authorization registration. In this case, the individual is the individual with the highest operation priority.

[0241] Step S5200, acquiring the image captured by the camera device 1500, detecting the individuals in the image, and obtaining the human body region image corresponding to each individual, and then executing step S5300.

[0242] In step S5200, human figure detection can be performed on the image using a pre-trained human figure detection model to obtain the individuals in the image and the image position coordinates of each individual in the image. In this way, the human body area image corresponding to each individual can be obtained based on the image position coordinates of each individual in the image.

[0243] In step S5200, when there is no individual in the image, the electronic device will continue to execute step S5200 to obtain images that continue to be captured for human detection.

[0244] Step S5300, performing face detection on the human body region image to obtain a facial region image of the corresponding individual, and then executing step S5400.

[0245] After obtaining the human body region image, face detection can be performed on the human body region image using a pre-trained face detection model to obtain image position coordinates of the face on the human body region image, and further obtain the face region image of the corresponding individual.

[0246] After obtaining the facial area image, proceed to step S5400 to perform identity authentication.

[0247] Step S5400, identity authentication is performed to determine the current responding individual, and when the current responding individual is determined, step S5500 is executed.

[0248] In step S5400, each facial region image obtained in step S5300 can be matched with the facial image input during the initial setting in step S5100 to determine whether the individual corresponding to each facial region image is an authorized individual. In the case where there is one authorized individual, the individual is the individual with the highest manipulation priority and can be determined as the current responding individual. In the case where there are two or more (including two) authorized individuals, the individual with the highest manipulation priority is screened out according to the authorization information of these authorized individuals as the current responding individual.

[0249] In step S5400, when the individuals corresponding to the facial area images obtained in step S5300 are not authorized individuals, the process returns to step S5200 to obtain further captured images for human detection.

[0250] Step S5500, starting from the image of step S5200, tracks and detects the current responding individual in the image sequence captured by the camera device 1500. For any frame image in the image sequence, it is determined whether the current responding individual is detected, that is, whether the tracking is successful. If the tracking is successful, step S5600 is executed to obtain a partial image corresponding to the upper body of the current responding individual. If the tracking fails, it returns to step S5200 to obtain the images that continue to be captured for human detection.

[0251] Step S5600, according to the image position coordinates of the image of the current responding individual in the image sequence, the area where the upper body of the individual is located is obtained, and then the partial image corresponding to the upper body of the individual is obtained.

[0252] The area where the upper body of the individual is located is also the gesture calculation area.

[0253] Step S5700, performing hand detection on the partial image obtained in step S5600 to obtain a hand area image corresponding to the hand area.

[0254] In step S5700, the hand region corresponding to the hand features in the local image can be extracted by using a pre-trained hand detection model, thereby obtaining a hand region image.

[0255] In step S5700, if a human hand is detected, step S5800 is executed to classify the human hand; if no human hand is detected, the process returns to step S5500 to continue individual tracking.

[0256] Step S5800, performing hand classification on the obtained hand area image to obtain the gesture state of the current responding individual.

[0257] Based on the hand area image obtained in step S5700, the gesture state of the current responding individual is judged. When the gesture state obtained by classification is a preset gesture, step S5900 is executed to perform a control response; when the gesture state obtained by classification is not a preset gesture, return to step S5500 and continue individual tracking.

[0258] Step S5900, controlling the electronic device to respond according to the control instruction corresponding to the gesture state obtained in step S5800.

[0259] For example, the gesture state of "OK" represents "play", and for another example, the gesture state of "palm facing the electronic device" represents "pause", etc.

[0260] The electronic device responds according to the gesture state acquired in step S5800. For example, when the electronic device 1000 is in the state of pausing video playback and the electronic device tracks that the current responding individual makes an "OK" gesture, the electronic device will respond by continuing to play the video.

[0261] According to the above examples, when the electronic device responds to the gesture control of an individual, an identity authentication step is added to determine the current responding individual based on the authorization information, and after determining the current responding individual, the individual is continuously tracked to respond only to the gesture state of the individual until the individual cannot be tracked, and then the current responding individual is re-determined. Therefore, the method of this example can achieve the purpose of responding to a specific individual, and this can also avoid the response confusion problem that may be caused by multiple people operating at the same time.

[0262] According to the above example, by using individual tracking technology, after the current responding individual is determined through step S5400, only steps S5500-S5900 need to be executed for individual tracking, eliminating steps S5200-S5400, thereby eliminating steps such as identity authentication and improving response speed.

[0263] In addition, according to the above examples, through a progressive mechanism of "human body detection-face detection-hand detection", the hand gesture state is tracked and detected from large to small. Compared with the existing technology, it can be recognized more accurately at the same distance, thereby achieving gesture recognition and control at a longer distance.

[0264] <Device Example>

[0265] This embodiment provides a control device for an electronic device, such as Figure 6 As shown, the control device 6000 of the electronic device may include an image acquisition module 6100 , a tracking detection module 6200 , a gesture recognition module 6300 and a manipulation execution module 6400 .

[0266] The image acquisition module 6100 is used to acquire a first image captured by the camera device 1500 .

[0267] The tracking detection module 6200 is used to detect a specific current responding individual in the first image.

[0268] The gesture recognition module 6300 is used to obtain the gesture state of the current responding individual according to the first image when the current responding individual is detected; and

[0269] The manipulation execution module 6400 is used to obtain the manipulation instruction currently issued by the responding individual to the electronic device 1000 according to the gesture state as a valid manipulation instruction.

[0270] In one embodiment, the image acquisition module 6100 is further used to acquire a second image captured by the camera device 1500, wherein the shooting time of the second image is earlier than the shooting time of the first image. In this embodiment, the control device 6000 also includes an individual determination module, which is used to detect an authorized individual in the second image; and, when an authorized individual is detected, select an individual with the highest control priority from the authorized individuals as the current response individual.

[0271] In one embodiment, when the individual determination module detects an authorized individual in a second image, it can be used to: detect individuals present in the second image and obtain a human body area image corresponding to each individual; obtain a facial area image of the corresponding individual based on the human body area image; and, based on the facial area image, detect whether the corresponding individual is an authorized individual.

[0272] In one embodiment, when the individual determination module detects individuals existing in the second image and obtains the human body area image corresponding to each individual, it can be used to: input the second image into a preset human figure detection model to obtain the human body area corresponding to each individual in the second image; and, based on the human body area, obtain the corresponding human body area image.

[0273] In one embodiment, when the individual determination module obtains the facial area image of the corresponding individual based on the human body area image, it can be used to: input the human body area image into a preset face detection model to obtain the facial area in the human body area image; and, based on the facial area, obtain the corresponding facial area image.

[0274] In one embodiment, the individual determination module can be used to: obtain a picked-up sound signal before the tracking detection module 6200 detects a specific current responding individual in the image; detect the existence of an authorized individual based on the sound signal; and, when an authorized individual is detected, select the individual with the highest manipulation priority among the detected authorized individuals as the current responding individual.

[0275] In one embodiment, the control device 6000 may further include a priority determination module, which is used to determine the current manipulation priority of the authorized individual. When determining the current manipulation priority of the authorized individual, the priority determination module may be used to: detect whether one of the events that trigger the priority change is occurred according to at least one event that is preset; when an event is detected, obtain a priority setting scheme corresponding to the event; and determine the current manipulation priority of the authorized individual according to the obtained priority setting scheme.

[0276] In one embodiment, when the gesture recognition module 6300 obtains the control instruction issued by the current responding individual to the electronic device according to the gesture state, it can be used to: when the gesture state belongs to a set gesture, obtain the control instruction corresponding to the set gesture as the control instruction issued by the current responding individual to the electronic device 1000.

[0277] In one embodiment, when the gesture recognition module 6300 obtains the gesture state of the current responding individual based on the first image, it can be used to: obtain a human body area image corresponding to the current responding individual based on the human body area corresponding to the current responding individual in the first image; and obtain the gesture state of the current responding individual based on the human body area image.

[0278] In one embodiment, when the gesture recognition module 6300 obtains the gesture state of the current responding individual based on the human body area image, it can be used to: obtain a partial image corresponding to the upper body of the current responding individual based on the human body area image; and, obtain the gesture state of the current responding individual based on the partial image.

[0279] In one embodiment, when the gesture recognition module 6300 obtains the gesture state of the current responding individual based on the partial image, it can be used to include: extracting the human hand area corresponding to the human hand features in the partial image to obtain a human hand area image; and, based on the human hand area image, determining the gesture state of the current responding individual.

[0280] In one embodiment, when the gesture recognition module 6300 determines the gesture state of the current responding individual based on the human hand area image, it can be used to: input the human hand area image into a preset classification model, wherein the classification model reflects the mapping relationship between the human hand image and the classification result of the gesture; and, based on the classification result output by the classification model, obtain the gesture state of the current responding individual.

[0281] In one embodiment, the control device 6000 may further include a classification model generation module, which is used to obtain the classification model. When obtaining the classification model, the classification model generation module may be used to: obtain a human hand image sample, wherein the human hand image sample has a label related to the corresponding gesture state; train the model parameters of the selected model through the human hand image sample; configure the selected model according to the model parameters obtained by training, and obtain the classification model.

[0282] In one embodiment, when the tracking detection module 6200 detects a specific current responding individual in the first image, it can be used to: obtain optical flow information between the first image and a previous image, wherein the previous image is an image that is taken earlier than the first image and in which the current responding individual is detected; and detect the current responding individual in the first image based on the position coordinates of the current responding individual in the previous image and the optical flow information.

[0283] In one embodiment, when the tracking detection module 6200 detects a specific current responding individual in the first image, it can be used to: extract pixel features of the current responding individual in a previous image captured by the camera device 1500, wherein the previous image is an image captured earlier than the first image and in which the current responding individual is detected; and, based on the pixel features, detect the current responding individual in the first image.

[0284] <Equipment Embodiment>

[0285] Figure 7 FIG. 1 is a schematic diagram of the hardware structure of an electronic device according to an embodiment. Figure 7 As shown, the electronic device 7000 includes a memory 7200, a processor 7100 and a camera 7500. The memory 7200 is used to store executable instructions. The processor 7100 is used to execute the control method according to the first embodiment of the method of the present disclosure or the second embodiment of the method of the present disclosure according to the control of the executable computer program.

[0286] The camera device 7500 is connected to the processor 7100 to output the captured image to the processor 7100 .

[0287] The electronic device 7000 can be any type of device such as a television, a projector, or a speaker.

[0288] like Figure 7 As shown, the electronic device 7000 may further include a display device 7600 .

[0289] Since the electronic device 7000 can be well adapted to long-distance gesture control, the display device 7600 can have a larger screen, for example, the screen can be greater than or equal to 50 inches.

[0290] The electronic device 7000 may be Figure 2 The electronic device 1000 shown may also be a device with other hardware structures, which is not limited here.

[0291] In another embodiment, the electronic device 7000 may include the above control device 6000 .

[0292] In one embodiment, each module of the above control device 6000 can be implemented by the processor 7100 running the computer instructions stored in the memory 7200.

[0293] <Medium Example>

[0294] In this embodiment, a computer-readable storage medium is also provided, which stores a computer program that can be read and executed by a computer. When the computer program is read and executed by the computer, it is used to execute a control method such as any of the above method embodiments 1 or method embodiments 2 of the present invention.

[0295] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing the processor 1100 to implement various aspects of the present invention.

[0296] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0297] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0298] The computer program instructions for performing the operation of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present invention.

[0299] Various aspects of the present invention are described herein with reference to the flow charts and / or block diagrams of the methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each box of the flow chart and / or block diagram and the combination of each box in the flow chart and / or block diagram can be implemented by computer-readable program instructions.

[0300] These computer-readable program instructions can be provided to a processor 1100 of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor 1100 of the computer or other programmable data processing device, a device for implementing the functions / actions specified in one or more blocks in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0301] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0302] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a part of a module, a program segment or an instruction, and a part of the module, a program segment or an instruction contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that it is equivalent to implement it by hardware, implement it by software, and implement it by combining software and hardware.

[0303] Embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the marketplace, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.

Claims

1. A method for controlling an electronic device, wherein the electronic device has a camera device, the method comprising: Acquire a second image captured by the camera device; detecting an authorized individual in the second image; In the case where an authorized individual is detected, an individual with the highest manipulation priority is selected from the detected authorized individuals as the current response individual; Acquire a first image captured by the camera device; detecting the current responding individual in the first image; in; The shooting time of the second image is earlier than the shooting time of the first image; When the current responding individual is detected, obtaining a gesture state of the current responding individual according to the first image; According to the gesture state, obtaining a control instruction issued by the current responding individual to the electronic device as a valid control instruction; The detecting the current responding individual in the first image comprises: Acquire optical flow information between the first image and a previous image, wherein the previous image is an image captured earlier than the first image and in which the current responding individual is detected; The current responding individual is detected in the first image according to the image position coordinates of the current responding individual in the previous image and the optical flow information.

2. The method according to claim 1, wherein: The detecting the authorized individual in the second image comprises: Detecting individuals in the second image, and obtaining a human body region image corresponding to each individual; Obtaining a facial region image of a corresponding individual according to the human region image; Based on the facial region image, it is detected whether the corresponding individual is an authorized individual.

3. The method according to claim 2, wherein: The detecting the individuals in the second image and obtaining a human body region image corresponding to each individual includes: Inputting the second image into a preset human figure detection model to obtain a human body region corresponding to each individual in the second image; According to the human body region, obtaining a corresponding human body region image; and / or, The step of obtaining a facial region image of a corresponding individual according to the human region image comprises: Inputting the human body region image into a preset face detection model to obtain a face region in the human body region image; According to the facial region, a corresponding facial region image is obtained.

4. The method according to claim 1, wherein: Prior to the step of detecting a specific current responding individual in the image, the method further comprises: Acquire the picked-up sound signal; Detecting the presence of an authorized individual based on the sound signal; In the case where an authorized individual is detected, an individual with the highest manipulation priority is selected from the detected authorized individuals as the current responding individual.

5. The method according to claim 1 or 4, wherein: Steps to determine the current control priority of authorized individuals include: According to at least one event preset to trigger a priority change, detecting whether one of the events occurs; In case an event is detected, obtaining a priority setting scheme corresponding to the event; According to the acquired priority setting scheme, the current control priority of the authorized individual is determined.

6. The method according to claim 5, wherein: The at least one event triggering the priority change includes at least one of an event in which the electronic device is in a specified application scenario, an event in which the electronic device receives specific voice content, and an event in which the electronic device senses a specific action.

7. The method according to claim 1, wherein: The obtaining, according to the gesture state, a control instruction issued by the current responding individual to the electronic device includes: In the case that the gesture state belongs to a set gesture, a manipulation instruction corresponding to the set gesture is acquired as the manipulation instruction issued by the current responding individual to the electronic device.

8. The method according to claim 1, wherein: The obtaining the gesture state of the current responding individual according to the first image includes: Obtaining a human body region image corresponding to the current responding individual according to the human body region corresponding to the current responding individual in the first image; The gesture state of the current responding individual is obtained according to the human body region image.

9. The method according to claim 8, wherein: The obtaining, according to the human body region image, the gesture state of the current responding individual includes: According to the human body region image, obtaining a partial image corresponding to the upper body of the current responding individual; The gesture state of the current responding individual is obtained according to the partial image.

10. The method according to claim 9, wherein: The obtaining, according to the partial image, the gesture state of the current responding individual includes: Extracting a human hand region corresponding to a human hand feature in the partial image to obtain a human hand region image; The gesture state of the current responding individual is determined according to the human hand area image.

11. The method according to claim 10, wherein: The step of determining the gesture state of the current responding individual according to the human hand region image includes: Inputting the human hand region image into a preset classification model, wherein the classification model reflects a mapping relationship between the human hand image and a classification result about the gesture; According to the classification result output by the classification model, the gesture state of the current responding individual is obtained.

12. The method according to claim 11, wherein: The step of obtaining the classification model comprises: Acquire a human hand image sample, wherein the human hand image sample has a label related to a corresponding gesture state; Training the model parameters of the selected model using the human hand image samples; The selected model is configured according to the model parameters obtained through training to obtain the classification model.

13. The method according to claim 1, wherein: The detecting a specific current responding individual in the first image comprises: Extracting pixel features of the current responding individual in a preceding image captured by the camera device, wherein the preceding image is an image captured earlier than the first image and in which the current responding individual is detected; The current responding individual is detected in the first image according to the pixel features.

14. A control method of an electronic device, the electronic device comprising a camera device, the control method comprising: detecting an authorized individual in an image captured by the camera device; In the case where an authorized individual is detected, an individual with the highest control priority is selected from the authorized individuals as the current responding individual, and the image is used as the first frame image; Tracking and detecting a gesture state of a specific current responding individual in a sequence of images captured by the camera device; According to the gesture state, obtaining a control instruction issued by the current responding individual to the electronic device; According to the control instruction, controlling the electronic device to respond; The first frame image is the second image, each frame image after the first frame image in the image sequence is the first image, and tracking and detecting the gesture state of a specific current responding individual in a sequence of images captured by the camera device includes: Acquire a first image in a sequence of images captured by the camera device; Detecting the current responding individual in the first image; wherein the shooting time of the second image is earlier than the shooting time of the first image; When the current responding individual is detected, obtaining a gesture state of the current responding individual according to the first image; The detecting the current responding individual in the first image comprises: Acquire optical flow information between the first image and a previous image, wherein the previous image is an image captured earlier than the first image and in which the current responding individual is detected; The current responding individual is detected in the first image according to the image position coordinates of the current responding individual in the previous image and the optical flow information.

15. The method according to claim 14, wherein: The method further comprises: In the case that the current responding individual is not detected in the subsequent image, taking the subsequent image as the first image, and executing the step of detecting the authorized individual in the first image captured by the camera device; The shooting time of the subsequent image is later than the shooting time of the first frame image in the image sequence.

16. A control device for an electronic device, comprising: An image acquisition module, used to acquire a second image captured by the camera device; an individual determination module, configured to detect an authorized individual in the second image; and in the case where an authorized individual is detected, selecting an individual with the highest manipulation priority among the detected authorized individuals as the current response individual; An image acquisition module, used to acquire a first image captured by the camera device; A tracking detection module, used for detecting a specific current responding individual in the first image; wherein the shooting time of the second image is earlier than the shooting time of the first image; a gesture recognition module, configured to obtain a gesture state of the current responding individual according to the first image when the current responding individual is detected; and A manipulation execution module, used for obtaining, according to the gesture state, a manipulation instruction issued by the current responding individual to the electronic device as a valid manipulation instruction; The tracking and detection module is specifically used to obtain optical flow information between the first image and a previous image, wherein the previous image is an image that is taken earlier than the first image and in which the current responding individual is detected; and, based on the position coordinates of the current responding individual in the previous image and the optical flow information, the current responding individual is detected in the first image.

17. An electronic device, comprising a camera device, the electronic device further comprising: A memory for storing executable instructions; A processor, configured to execute the control method according to any one of claims 1 to 15 under the control of the executable computer program; Wherein, the camera device is connected to the processor to output the captured image to the processor. 18 . The electronic device according to claim 17 , further comprising a display device, wherein a screen of the display device is greater than or equal to 50 inches.

19. A computer-readable storage medium, wherein: The computer-readable storage medium stores a computer program that can be read and executed by a computer, and the computer program is used to execute the control method according to any one of claims 1 to 15 when being read and executed by the computer.

Images