Graphical programming method based on face recognition

Abstract

The present invention discloses a graphical programming method and device based on face recognition. The method comprises: after detecting that a face recognition instruction block is dragged to a programming area in a graphical programming interface, calling a webpage real-time communication interface of a browser to obtain data captured by a camera, wherein the face recognition instruction block is a programming instruction block related to face recognition; based on the data captured by the camera, using a face recognition module to perform face detection and recognition in real time; based on the results of the face detection and recognition, controlling the actions of the stage characters in the stage area in the graphical programming interface, and simultaneously displaying the face image captured by the camera in the stage area. The present invention solves the technical problem that face recognition is relatively complicated during graphical programming.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a graphical programming method based on face recognition. Background Art

[0002] With the rapid development of artificial intelligence and computer vision technology, face recognition technology has been widely used in many fields such as security monitoring, identity authentication, and smart home. However, existing face recognition systems usually face high barriers to use, which are mainly reflected in the following aspects. First, these systems usually require users to have high programming skills, making it difficult for non-professional users to use them effectively. Second, traditional face recognition systems usually require complex hardware and software configurations, which further increases the difficulty of using the system. Finally, the existing interaction methods are mostly command lines or code interfaces, which are not intuitive enough, resulting in a poor user experience. These shortcomings limit the popularity and application of face recognition technology among a wider population.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiment of the present invention provides a graphical programming method based on face recognition, so as to at least solve the technical problem that face recognition is relatively complicated during graphical programming.

[0005] According to one aspect of an embodiment of the present invention, a graphical programming method based on face recognition is provided, comprising: after detecting that a face recognition instruction block is dragged to a programming area in a graphical programming interface, calling a web real-time communication interface of a browser to obtain data captured by a camera, wherein the face recognition instruction block is a programming instruction block related to face recognition; based on the data captured by the camera, performing face detection and recognition in real time using a face recognition module; based on the results of the face detection and recognition, controlling the actions of stage characters in a stage area in the graphical programming interface, and simultaneously displaying the face image captured by the camera in the stage area.

[0006] According to another aspect of an embodiment of the present invention, a graphical programming device based on face recognition is also provided, including: a calling module, configured to call the WebRTC interface of the browser to obtain data captured by a camera after detecting that a face recognition instruction block is dragged to a programming area in a graphical programming interface, wherein the face recognition instruction block is a programming instruction block related to face recognition; a recognition module, configured to perform face detection and recognition in real time using a face recognition module based on the data captured by the camera; an interaction module, configured to control the actions of stage characters in a stage area in the graphical programming interface based on the results of the face detection and recognition, and simultaneously display the face image captured by the camera in the stage area.

[0007] In an embodiment of the present invention, after detecting that a face recognition instruction block is dragged to a programming area in a graphical programming interface, the webpage real-time communication interface of the browser is called to obtain data captured by the camera, wherein the face recognition instruction block is a programming instruction block related to face recognition; based on the data captured by the camera, a face recognition module is used to perform face detection and recognition in real time; based on the results of the face detection and recognition, the actions of the stage characters in the stage area in the graphical programming interface are controlled, and the face image captured by the camera is displayed in the stage area at the same time. Through the above scheme, the technical problem of complex face recognition in graphical programming is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0009] Figure 1 is a flow chart of a graphical programming method based on face recognition according to an embodiment of the present invention;

[0010] Figure 2 is a flowchart of a deep learning model training method according to an embodiment of the present invention;

[0011] Figure 3 is an architectural diagram of a graphical programming system in a Web browser according to an embodiment of the present invention;

[0012] Figure 4 is a graphical programming interface according to an embodiment of the present invention;

[0013] Figure 5 This is a display interface diagram of opening a camera building block box according to an embodiment of the present invention;

[0014] Figure 6This is an interface diagram of a face recognition instruction building block expanded in a code area according to an embodiment of the present invention;

[0015] Figure 7 is a flowchart of another graphical programming method based on face recognition according to an embodiment of the present invention;

[0016] Figure 8 is an interface diagram of a graphical code displayed in a code area according to an embodiment of the present invention;

[0017] Fig. 9 is a flowchart of another method for training a deep learning model for face recognition according to an embodiment of the present invention;

[0018] Fig.10 is a structural diagram of a graphical programming device based on face recognition according to an embodiment of the present invention;

[0019] Fig.11 A schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0020] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0021] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0022] According to an embodiment of the present invention, a method embodiment of a graphical programming method based on face recognition is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0023] Figure 1 is a graphical programming method based on face recognition according to an embodiment of the present invention, such as Figure 1 As shown, the method comprises the following steps:

[0024] Step S102, after detecting that a face recognition instruction block is dragged to a programming area in a graphical programming interface, calling a web real-time communication interface of a browser to obtain data captured by a camera, wherein the face recognition instruction block is a programming instruction block related to face recognition.

[0025] After detecting that the face recognition instruction block is dragged to the programming area in the graphical programming interface, the WebRTC interface of the browser is called to obtain the data captured by the camera.

[0026] Step S104: Based on the data captured by the camera, a face recognition module is used to perform face detection and recognition in real time.

[0027] The data captured by the camera is preprocessed, wherein the preprocessing includes at least one of the following: grayscale, scale transformation; based on the preprocessed data captured by the camera, a face recognition module is used to perform face detection and recognition in real time to obtain the face detection and recognition results. Wherein, the face detection and recognition in real time using the face recognition module includes at least one of the following: gender recognition, age recognition, face shape recognition, emotion recognition, accessories recognition, and eye state recognition. The grayscale and scale transformation in the preprocessing step of the present application effectively reduce the complexity of the data, so that the subsequent face recognition algorithm can process and analyze image information more quickly and reduce the consumption of computing resources. In addition, through the above steps, the application breadth and flexibility of the system are also improved. The present application can not only provide users with rich and accurate personalized services, but also maintain stable recognition performance in complex and changing environments, and has extremely high practical value and market competitiveness.

[0028] In some embodiments, the face recognition module may be a deep learning model for face recognition, and the training method of the deep learning model may be as follows: Figure 2 As shown, the following steps are included:

[0029] Step S1042: Acquire a training data set, and preprocess the training data in the training data set.

[0030] The preprocessing includes at least one of the following: data cleaning, standardization and data enhancement; the data enhancement includes generating diversified face images through a generative adversarial network to enhance the training data set.

[0031] Step S1044, extracting multi-layer features from the preprocessed training data using a convolutional neural network;

[0032] Step S1046, based on the extracted multi-layer features, a deep learning model with a dual attention mechanism is used to perform face recognition to obtain a predicted recognition result;

[0033] Step S1048, compare the predicted recognition result and the true value to obtain the loss of the deep learning model, and use the loss to update the model parameters of the deep learning model.

[0034] After the deep learning model is trained, the method further includes: adding noise to the model parameters of the deep learning model so that the predicted recognition result output by the deep learning model satisfies the differential privacy constraint.

[0035] Step S106, based on the results of the face detection and recognition, controlling the actions of the stage characters in the stage area in the graphical programming interface, and simultaneously displaying the face image captured by the camera in the stage area.

[0036] Firstly, based on the results of the face detection and recognition, the actions of the stage characters in the stage area in the graphical programming interface are controlled.

[0037] At the same time, the face image captured by the camera is displayed in the stage area. A video element is created based on the data captured by the camera, and the video corresponding to the data captured by the camera is overlaid on the stage in a transparent form; and / or, the data captured by the camera is placed in a canvas, and after intercepting the current frame of the data captured by the camera, the corresponding image data is generated using the canvas, and the image data is placed in the stage area, so that the stage background of the stage area is located at the bottom layer, the video corresponding to the data captured by the camera is located above the stage background, and the stage character is located above the video corresponding to the data captured by the camera.

[0038] This application introduces a graphical programming platform, so that users can implement face recognition functions without professional programming skills, which greatly reduces the technical threshold and is convenient for ordinary users to use. Secondly, the method of using a web browser to call the camera simplifies the hardware and software configuration, reduces the complex face recognition steps in the traditional system, and thus improves the usability of the system. Finally, this application uses an intuitive graphical building block instruction interface to replace the traditional command line or code interface, greatly improving the friendliness of user interaction and the user experience. These improvements enable face recognition technology to be more widely used in various scenarios, providing users with convenient and efficient solutions.

[0039] The present application provides a technical solution based on a graphical programming platform in a web browser, which calls a web camera for face recognition through building block instructions, so that users can realize the face recognition function through simple graphical building block instructions. Figure 3 is an architectural diagram of a graphical programming system in a Web browser according to an embodiment of the present application, such as Figure 3 As shown, the system includes a front-end interface 12, a back-end engine 14 and a model library 16. The front-end interface 12 is a graphical programming interface for user operation interaction. The back-end engine 14 is used to process front-end user instructions. The model library 16 contains multiple models.

[0040] The front-end interface 12 is a graphical programming interface for user operation interaction. The graphical programming interface provides an intuitive and user-friendly way for users to create and edit programs by dragging and dropping programming instruction blocks. In addition, the front-end interface 12 is also responsible for real-time feedback of user operations and displaying the status and results of program execution.

[0041] The backend engine 14 is used to process frontend user instructions. Specifically, it is used to receive programming instructions from the frontend interface 12, parse these instructions, and convert them into executable code or instruction sets. The backend engine 14 also includes a runtime environment in which user-defined programs can be executed and data and events generated during program execution can be processed. In addition, the backend engine 14 can also interact with external systems and services to achieve more complex functions and tasks.

[0042] The model library 16 includes a variety of models, which can be used for different programming tasks and application scenarios. Each model in the model library 16 is pre-trained and optimized and can be directly applied to the user's programming project. These models can include machine learning models such as the face recognition model, data analysis model, simulation model, etc. provided in this application. Users can select a suitable model from the model library 16 and integrate it into their own program to achieve specific functions and goals. Users can also generate and train models through a graphical programming interface and store the trained models in the model library 16. In addition, the model library 16 also supports the update and expansion of the model, and users can add new models or update existing models as needed to maintain the advancement and flexibility of the system.

[0043] Specifically, the front-end interface 12 is a graphical programming interface, for example, Figure 4 As shown, it includes a programming area 32 (also called a console) for editing graphical codes, a stage area 36 for displaying the execution effect of the graphical codes, and a character bar 34 for providing the stage area 36 with stage characters and backgrounds.

[0044] The programming area 32 is the main operation area for users to perform graphical programming, and includes two components. The first is the code area 322, which is used to display and edit graphical codes. Users can create and modify program logic in this area by dragging and connecting programming instruction blocks. The second is the programming instruction building block box 324, which provides various programming instruction blocks. Users can drag these blocks into the code area 322 to build a program. The building block box contains multiple categories of building blocks, such as the camera building block box 3242, which covers various programming needs from basic operations to advanced functions.

[0045] The stage area 36 is used to display the execution effect of the graphical code. The user can directly observe the program running results by clicking the start button of the stage area 36. Whether it is animation, interaction or data display, the stage area 36 provides a visual platform so that users can instantly see the actual effect of the code they wrote. This is very important for debugging and improving the program, because users can find and correct errors by observing the output effect.

[0046] Figure 5 Click on the display interface of the camera building block box, such as Figure 5As shown, by clicking the camera block box 3242, a plurality of face recognition instruction blocks will appear. These programming instruction blocks provide a wealth of functions, and users can use these blocks to easily control the camera and process images. Such as "recognize face", "touch face", "move to face", "face facing", "face coordinates", "face size, width and height", "face tilt angle" and "face tilt left / right". These face recognition instruction blocks enable users to control and track facial features in detail, thereby realizing applications such as face detection and expression recognition.

[0047] When users drag these face recognition instruction blocks to the code area, some blocks will automatically expand to more detailed options. Figure 6 As shown in the figure, when the user drags the face recognition instruction block "Recognize face" to the code area, the block will expand to include multiple options such as "Top of head", "Left eye", "Right eye", "Nose", "Mouth", "Chin", "Hand", etc. After these face recognition instruction blocks are dragged to the code area, they can be combined with other blocks to form complex program logic. For example, the user can create a program that performs a specific action or triggers a specific event when the "left eye" is detected. This graphical programming method not only simplifies the programming process, but also greatly lowers the threshold of programming, allowing more people to easily get started and experience the fun of programming.

[0048] The present application also provides another graphical programming method based on face recognition, such as Figure 7 As shown, the method comprises the following steps:

[0049] Step S702, initializing the graphical programming platform.

[0050] The user opens the graphical programming platform, which provides programming instruction blocks for the user to choose from. Figure 4 As shown, users can build the required program by dragging and dropping building block instructions.

[0051] Step S704, drag the face recognition instruction block.

[0052] The user can select and drag the face recognition instruction blocks in the programming interface. The camera block box includes instruction blocks for turning on / off the camera and face recognition instruction blocks. The user can drag these instructions to the code area and combine them with other blocks to form a complete program flow, for example Figure 8 shown.

[0053] Step S706, start the program.

[0054] When the user clicks the "Run" button on the interface, the system starts to execute the program created by the user. At this time, the platform will execute each instruction block in sequence according to the user's programming logic.

[0055] Step S708, calling the camera to obtain the video stream.

[0056] Since the program contains face recognition instruction blocks, the system calls the built-in WebRTC interface of the browser through JavaScript code to start the camera and obtain the real-time video stream. The obtained video stream will be used as input data for subsequent face recognition processing.

[0057] The video stream is processed using the face recognition model. The acquired video stream data is passed to the face recognition model. The face recognition model detects and recognizes faces in the video stream. The recognition content may include but is not limited to gender recognition, age recognition, face shape recognition, emotion recognition, accessories recognition, and eye state recognition.

[0058] The training process of the face recognition model will be described in detail below and will not be repeated here.

[0059] Step S710, display the recognition result.

[0060] The recognition results are returned to the graphical programming platform, which displays them in a visual way on the user interface. Users can further adjust the program logic or perform other operations based on the recognition results.

[0061] Specifically, the image captured by the camera is displayed on the stage area. For example, a video element is created based on the data captured by the camera, and the video corresponding to the data captured by the camera is overlaid on the stage in a transparent form; and / or the data captured by the camera is placed in a canvas, and after intercepting the current frame of the data captured by the camera, the corresponding image data is generated using the canvas, and the image data is placed in the stage area, so that the stage background of the stage area is located at the bottom layer, the video corresponding to the data captured by the camera is located above the stage background, and the stage character is located above the video corresponding to the data captured by the camera.

[0062] Control the stage character based on the recognition result. For example, Figure 8 In the program, when the left eye is recognized from the image collected by the camera using the face recognition model, the stage character is controlled to rotate 30 degrees counterclockwise; when the right eye is recognized from the image collected by the camera using the face recognition model, the stage character is controlled to rotate 30 degrees clockwise.

[0063] In addition, the platform supports users to adjust the program in real time. Users can drag new instruction blocks during the program running and re-run the program to obtain different recognition effects. At the same time, the platform will provide timely feedback and suggestions based on user operations and recognition results to help users optimize program design.

[0064] Through the above method, the present invention realizes the face recognition function based on the graphical programming platform, so that users can complete complex face recognition tasks without writing codes, which greatly reduces the technical threshold of programming.

[0065] The embodiment of the present invention also provides another training method for a deep learning model for face recognition, which aims to improve the accuracy, robustness and data privacy protection capabilities of the face recognition model by combining a generative adversarial network (GAN), a convolutional neural network (CNN), a dual attention mechanism and differential privacy technology. Fig. 9 As shown, the method includes:

[0066] Step S902: data preprocessing.

[0067] The data preprocessing method can be as follows:

[0068] First, the image is normalized. Normalize the image and scale the pixel values ​​to the range of [0,1]. Normalization can accelerate the convergence of the model. Then, perform data augmentation on the image. For example, randomly rotate the image by a certain angle; randomly enlarge or reduce the image; randomly translate the image horizontally or vertically; randomly cut some parts of the image; randomly flip the image horizontally or vertically; randomly adjust the brightness, contrast, and saturation of the image.

[0069] Then, the image is denoised. First, a Gaussian kernel is generated and normalized; then, the image is denoised based on the offset value of the pixel value at each pixel position of the image, the horizontal gradient of the image at each pixel position, the vertical gradient of the image at each pixel position, and the noise intensity parameter, wherein the noise intensity parameter is used to control the sensitivity to the difference in the pixel value of the image during the denoising process. Specifically, the following steps are included:

[0070] 1) Generate a Gaussian kernel and normalize the Gaussian kernel.

[0071] Assume that the image has been loaded as a matrix I, where I(i,j) represents the pixel value of the image at position (i,j). Select the size and standard deviation of the Gaussian kernel. Select the size of the Gaussian kernel (such as 3x3, 5x5, 7x7, etc.) and the standard deviation σ. The larger the Gaussian kernel size, the stronger the blurring effect, but more details may be lost at the same time. The larger the standard deviation σ, the more significant the blurring effect. Generate a Gaussian kernel. Generate a Gaussian kernel matrix K according to the Gaussian function formula. For a Gaussian kernel of size k×k, the center position is (k / / 2, k / / 2):

[0072]

[0073] in, σ is the standard deviation, which determines the width of the Gaussian distribution, (x,y) is the coordinate of the pixel in the image relative to the center of the kernel, and k is the size of the Gaussian kernel.

[0074] Then, normalize the Gaussian kernel. In order to ensure that the weighted sum of the Gaussian kernel is 1, it needs to be normalized:

[0075]

[0076] 2) Perform denoising.

[0077] Multi-scale Gaussian blur is achieved by applying multiple Gaussian kernels of different scales. The standard deviation σ of each Gaussian kernel is different, which can capture the image features at different scales, thereby improving the denoising effect. The improved denoising processing formula is:

[0078]

[0079] N=e -η·I(i+m ,j+n)-I(i ,j)

[0080] Among them, I(i,j) is the pixel value of the original image at position (i,j), m,n are the offsets of the horizontal and vertical coordinates relative to the center of the Gaussian kernel, tx is the horizontal gradient of the image at each pixel position, ty is the vertical gradient of the image at each pixel position, β is the gradient weight, and N is the noise intensity parameter, which is used to control the sensitivity of the pixel value difference of the image during the denoising process. It is obtained based on the noise intensity η, the pixel value of the image at position (i,j), and the offset value of the pixel value of the image at position (i,j).

[0081] This embodiment performs denoising on the image based on the pixel value offset, horizontal gradient, vertical gradient and noise intensity parameter at each pixel position, which can make the denoising process more detailed and accurate. Among them, the noise intensity parameter is used to control the sensitivity to the pixel value difference in the denoising process, which helps to retain more image details and edge information while effectively removing noise, making the denoised image clearer and more natural.

[0082] 3) Deal with boundary issues.

[0083] In the denoising process, if the pixels at the boundary do not have enough neighboring pixels for calculation, the boundary can be mirrored and expanded so that the pixels at the boundary have enough neighboring pixels. Another example is to fill them with a constant value (such as 0).

[0084] Step S904: feature extraction.

[0085] After data preprocessing, a convolutional neural network (CNN) is used to perform multi-layer feature extraction on the standardized image. Through the stacked structure of the deep convolutional network, more abstract and recognizable feature representations can be extracted. The mathematical representation of feature extraction is as follows: f(X) = CNN(X), where f(X) represents the features extracted by the convolutional network.

[0086] Step S906, training the recognition model.

[0087] In order to enhance the recognition ability of the model, the present invention introduces a dual attention mechanism, including a spatial attention mechanism and a channel attention mechanism. The spatial attention mechanism calculates the spatial attention weight S by focusing on the importance of different positions in the image, and the formula is as follows: S = softmax(Ws·f(X)).

[0088] In addition, the channel attention mechanism calculates the channel attention weight C by paying attention to the weights of different feature channels. The formula is as follows: C = softmax(Wc f(X))

[0089] Next, the spatial and channel attention mechanisms are combined to obtain the final feature representation F: F = S·C·f(X).

[0090] Among them, CNN(X) represents the operation of Convolutional Neural Network (CNN), which means the process of extracting features from input data X through CNN network. f(X) represents the feature representation of input image X after being extracted by Convolutional Neural Network. Ws represents the trainable weight matrix in spatial attention mechanism, which is used to calculate spatial attention weight S. S is the attention weight matrix in spatial attention mechanism, which represents the model's evaluation of the importance of different spatial positions in the input image. Wc is the trainable weight matrix in channel attention mechanism, which is used to calculate channel attention weight C. C is the attention weight matrix in channel attention mechanism, which represents the model's evaluation of the importance of different feature channels. F is the final feature representation matrix after combining spatial and channel attention mechanisms, which represents the comprehensive feature representation of input image X under dual attention mechanism.

[0091] The feature representation F obtained by the above dual attention mechanism is used for classification and recognition, and finally the prediction is output. The output prediction is compared with the true value to obtain the loss between the two, and the parameters of the model are updated based on the loss.

[0092] In order to protect the user's privacy data during the training process, the present invention introduces differential privacy technology. Each time the model parameters are updated, a noise N that satisfies the differential privacy constraint is added. The formula is as follows:

[0093] θ′=θ+N(0,σ 2 )

[0094] Among them, θ represents the original model parameters, N(0,σ 2 ) represents Gaussian noise with mean 0 and variance σ2. The introduction of this noise enables the model to effectively protect the privacy of training data when it is publicly released.

[0095] After the training is completed, a new unlabeled face image is input, and the trained model is used for prediction, and the recognition result and the corresponding confidence score are output. Through the above steps, the accuracy and robustness of the face recognition model can be significantly improved, especially in scenes with large changes in lighting, expression and posture. In addition, the use of differential privacy technology effectively protects user data and improves the practicality and security of the model.

[0096] An embodiment of the present invention also provides a face recognition method, which obtains an image to be recognized; inputs the image to be recognized into a deep learning model trained using the method described above to obtain a face recognition result.

[0097] The present application also provides a graphical programming device based on face recognition, such as Fig.10As shown, it includes: a calling module 12, which is configured to call the WebRTC interface of the browser to obtain data captured by the camera after detecting that the face recognition instruction block is dragged to the programming area in the graphical programming interface, wherein the face recognition instruction block is a programming instruction block related to face recognition; a recognition module 14, which is configured to perform face detection and recognition in real time using the face recognition module based on the data captured by the camera; an interaction module 16, which is configured to control the actions of the stage characters in the stage area in the graphical programming interface based on the results of the face detection and recognition, and simultaneously display the face image captured by the camera in the stage area.

[0098] It should be noted that: the graphical programming device based on face recognition provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the graphical programming device based on face recognition provided in the above embodiment and the graphical programming method embodiment based on face recognition belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0099] Fig.11 FIG. 1 shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present disclosure. It should be noted that: Fig.11 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0100] like Fig.11 As shown, the electronic device includes a central processing unit (CPU) 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage part 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for system operation are also stored. The CPU 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0101] The following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, etc.; an output section 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, a modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed, so that a computer program read therefrom is installed into the storage section 1008 as needed.

[0102] In particular, according to an embodiment of the present disclosure, the process described below with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1009, and / or installed from the removable medium 1011. When the computer program is executed by the central processing unit (CPU) 1001, various functions defined in the method and apparatus of the present application are executed. In some embodiments, the electronic device may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0103] It should be noted that the computer-readable medium shown in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0104] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0105] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, and the units described may also be arranged in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0106] As another aspect, the present application further provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device.

[0107] The computer-readable medium carries one or more programs, and when the one or more programs are executed by an electronic device, the electronic device implements the method described in the following embodiments. For example, the electronic device can implement each step of the method embodiment.

[0108] As another aspect, the present application further provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program executable by the at least one processor, and the computer program is used to enable the electronic device to perform the method of the embodiment of the present application when executed by the at least one processor.

[0109] As another aspect, the present application further provides a computer program product, including a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to execute the method of the embodiment of the present application.

[0110] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0111] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0112] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0113] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0114] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0115] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0116] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A graphical programming method based on face recognition, characterized in that: include: After detecting that a face recognition instruction block is dragged to a programming area in a graphical programming interface, calling a webpage real-time communication interface of a browser to obtain data captured by a camera, wherein the face recognition instruction block is a programming instruction block related to face recognition; Based on the data captured by the camera, using a face recognition module to perform face detection and recognition in real time; Based on the results of the face detection and recognition, the actions of the stage characters in the stage area in the graphical programming interface are controlled, and at the same time, the face images captured by the camera are displayed in the stage area.

2. The method according to claim 1, characterized in that Based on the data captured by the camera, a face recognition module is used to perform face detection and recognition in real time, including: Preprocessing the data captured by the camera, wherein the preprocessing includes at least one of the following: grayscale conversion and scale transformation; Based on the pre-processed data captured by the camera, a face recognition module is used to perform face detection and recognition in real time to obtain the face detection and recognition results.

3. The method according to claim 2, characterized in that Real-time face detection and recognition using the face recognition module includes at least one of the following: gender recognition, age recognition, face shape recognition, emotion recognition, accessories recognition, and eye state recognition.

4. The method according to claim 1, characterized in that: Displaying the face image collected by the camera in the stage area includes: Creating a video element based on the data captured by the camera, and overlaying the video corresponding to the data captured by the camera on the stage in a transparent form; and / or The data captured by the camera is placed into a canvas, and after intercepting a current frame of the data captured by the camera, corresponding image data is generated using the canvas, and the image data is placed into the stage area, so that the stage background of the stage area is located at the bottom layer, the video corresponding to the data captured by the camera is located above the stage background, and the stage character is located above the video corresponding to the data captured by the camera.

5. The method according to claim 1, characterized in that Calling the web real-time communication interface of the browser to obtain the data captured by the camera includes: calling the WebRTC interface of the browser to obtain the data captured by the camera.

6. The method according to claim 1, characterized in that The graphical programming interface includes the programming area for editing graphical code and the stage area for displaying the execution effect of the graphical code, wherein the programming area includes a code area for displaying the graphical code and a programming instruction building block box for providing programming instruction building blocks for the code area, wherein the programming instruction building block box includes a camera building block box, and the camera building block box is used to provide the face recognition instruction building blocks, gesture recognition instruction building blocks, camera call building blocks, and result display building blocks.

7. A graphical programming device based on face recognition, characterized in that: include: A calling module is configured to call a WebRTC interface of a browser to obtain data captured by a camera after detecting that a face recognition instruction block is dragged to a programming area in a graphical programming interface, wherein the face recognition instruction block is a programming instruction block related to face recognition; A recognition module configured to perform face detection and recognition in real time using a face recognition module based on data captured by the camera; The interaction module is configured to control the actions of the stage characters in the stage area in the graphical programming interface based on the results of the face detection and recognition, and simultaneously display the face image captured by the camera in the stage area.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 6.

9. A computer device, characterized in that: include: Memory and processor, The memory stores a computer program; The processor is used to execute the computer program stored in the memory, and when the computer program is run, the processor is enabled to execute the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images