AI graphical programming method and device based on image recognition

Abstract

The invention discloses an AI graphical programming method and device based on image recognition. The method comprises the following steps: in response to detection that a graphical code of a programming area in a graphical programming interface comprises a programming instruction building block related to a camera, acquiring an image in real time through the camera; based on the programming instruction building block related to the camera, the collected image is recognized to obtain an image recognition result, and a real-time instruction is generated based on the image recognition result; and based on the real-time instruction, controlling the action of a stage role in a stage area in the graphical programming interface. According to the method and the device, the technical problem that real-time interaction between the stage role and the real scene of the user cannot be realized in graphical programming is solved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to an AI graphical programming method and device based on image recognition. Background Art

[0002] In recent years, graphical programming platforms have been widely used in education and programming learning. These platforms have greatly lowered the threshold for programming by dragging building block instructions, making programming education more intuitive and easy to understand. The main functions of some existing graphical programming platforms include: graphical programming building block functions that allow users to create programs by dragging building blocks, stage control functions that allow users to create and control various characters to perform actions on the stage, and event-driven programming functions that trigger character or scene changes through events. Although these technical solutions have been widely used, they are still lacking in real-time interaction between stage characters and users' real scenes, especially interactive control based on faces and gestures.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present invention provide an AI graphical programming method and device based on image recognition, so as to at least solve the technical problem that the stage characters and the user's real scene cannot interact in real time in graphical programming.

[0005] According to one aspect of an embodiment of the present invention, there is provided a graphical programming method based on image recognition, comprising: in response to detecting that a graphical code in a programming area in a graphical programming interface includes programming instruction blocks related to a camera, acquiring an image in real time through a camera; based on the programming instruction blocks related to the camera, identifying the acquired image to obtain an image recognition result, and generating real-time instructions based on the image recognition result; and based on the real-time instructions, controlling the actions of a stage character in a stage area in the graphical programming interface.

[0006] According to another aspect of an embodiment of the present invention, a graphical programming device based on image recognition is also provided, including: an image acquisition module, configured to acquire images in real time through a camera in response to detecting that the graphical code in the programming area in the graphical programming interface includes programming instruction blocks related to the camera; a recognition module, configured to identify the acquired image based on the programming instruction blocks related to the camera to obtain an image recognition result, and generate real-time instructions based on the image recognition result; and a control module, configured to control the actions of the stage characters in the stage area in the graphical programming interface based on the real-time instructions.

[0007] In an embodiment of the present invention, in response to detecting that the graphical code of the programming area in the graphical programming interface includes programming instruction blocks related to the camera, the camera collects images in real time; based on the programming instruction blocks related to the camera, the collected images are identified to obtain image recognition results, and real-time instructions are generated based on the image recognition results; based on the real-time instructions, the actions of the stage characters in the stage area of ​​the graphical programming interface are controlled. Through the above solution, the technical problem that the stage characters in the graphical programming cannot interact with the user's real scene in real time is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0009] Figure 1 is a flow chart of a graphical programming method based on image recognition according to an embodiment of the present invention;

[0010] Figure 2 is an architectural diagram of a graphical programming system in a Web browser according to an embodiment of the present invention;

[0011] Figure 3 is an interface diagram of a graphical programming interface according to an embodiment of the present invention;

[0012] Figure 4 This is an interface diagram of a display interface of a camera building block box according to an embodiment of the present invention;

[0013] Figure 5 is an interface diagram of a programming instruction building block after expansion in a code area according to an embodiment of the present invention;

[0014] Figure 6 is a flow chart of another graphical programming method based on image recognition according to an embodiment of the present invention;

[0015] Figure 7 is a flow chart of a method for recognizing an image according to an embodiment of the present invention;

[0016] Figure 8 is a structural schematic diagram of a graphical programming device for image recognition according to an embodiment of the present invention;

[0017] Fig. 9 A schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0018] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0019] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0020] According to an embodiment of the present invention, a method embodiment of graphical programming based on image recognition is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0021] Figure 1 is a graphical programming method based on image recognition according to an embodiment of the present invention, such as Figure 1 As shown, the method comprises the following steps:

[0022] Step S102, in response to detecting that the graphical code in the programming area of ​​the graphical programming interface includes programming instruction blocks related to the camera, real-time image acquisition is performed through the camera.

[0023] Graphical programming is a programming method that uses visual elements (such as graphics, blocks, nodes, etc.) to represent programming logic and operations instead of traditional text code. With graphical programming, users can build programs by dragging and combining these visual elements such as programming instruction blocks.

[0024] Programming instruction blocks are represented in the form of blocks, each of which represents a command or a logical structure, such as conditional judgment, loop, etc. Users write programs by dragging these blocks. Graphical programming is event-driven. Users can create code blocks that are triggered when a certain event occurs. For example, "when a button is clicked" or "when a gesture is recognized." In the graphical programming environment, users can create and control characters (sprites) to perform various actions on the stage. Characters can move, rotate, change appearance, etc. Blocks can be connected and nested to form complex logical structures. For example, a conditional judgment block can be nested in a loop block.

[0025] In an embodiment of the present application, the graphical programming interface includes the programming area for editing the graphical code and the stage area for displaying the execution effect of the graphical code, wherein the programming area includes a code area for displaying the graphical code and a programming instruction building block box for providing programming instruction building blocks for the code area, wherein the programming instruction building block box includes a camera building block box, and the camera building block box is used to provide the programming instruction blocks related to the camera.

[0026] The programming instruction blocks related to the camera include at least one of the following: programming instruction blocks related to the attributes of the camera, wherein the programming instruction blocks related to the attributes of the camera include programming instruction blocks for controlling the camera switch and programming instruction blocks for controlling the transparency of the camera imaging; programming instruction blocks related to the recognition of images captured by the camera, wherein the programming instruction blocks related to the recognition of images include at least one of the following: programming instruction blocks related to the face of the target object recognized from the image and programming instruction blocks related to the gesture of the target object.

[0027] In response to detecting that one or more programming instruction blocks in the programming instruction block box are dragged from the programming instruction block box to the code area, a graphical code corresponding to the one or more programming instruction blocks is generated in the code area. In response to detecting that the graphical code in the programming area in the graphical programming interface includes programming instruction blocks related to the camera, an image is captured in real time through the camera.

[0028] Step S104, based on the programming instruction building blocks related to the camera, the collected image is identified to obtain an image recognition result, and a real-time instruction is generated based on the image recognition result.

[0029] In a case where the programming instruction block related to the camera is a programming instruction block related to the face of the target object, the face of the target object is identified, and it is determined whether the face of the target object meets the conditions indicated by the programming instruction block related to the face of the target object, so as to obtain the image recognition result of the face; in a case where the programming instruction block related to the camera is a programming instruction block related to the gesture of the target object, the gesture of the target object is identified, and it is determined whether the gesture of the target object meets the conditions indicated by the programming instruction block related to the gesture of the target object, so as to obtain the image recognition result of the gesture.

[0030] When the camera-related programming instruction block is a programming instruction block related to the face of the target object, the real-time instruction is generated based on the image recognition result of the face and the graphical code of the programming area; when the camera-related programming instruction block is a programming instruction block related to the gesture of the target object, the real-time instruction is generated based on the image recognition result of the gesture and the graphical code of the programming area.

[0031] This application combines facial gesture recognition technology to capture the user's gestures through the camera to generate corresponding instructions, achieving a more natural and intuitive interaction method. These diverse interaction modes not only improve the flexibility and practicality of the platform, but also greatly enrich the user experience, making the programming and control process more vivid and interesting. Combining these technical means, higher real-time performance and response speed can be achieved to meet users' needs for diversified interaction modes.

[0032] In other embodiments, real-time instructions can also be generated through touch screens, voice commands, etc. For example, through a touch screen, users can directly draw or drag objects on the screen to perform intuitive operations and command input, which not only simplifies the operation process but also improves the user's interactive experience. Through voice commands, users can use natural language to directly control characters and objects in the platform, which makes the operation more convenient and efficient.

[0033] Step S106, controlling the actions of the stage characters in the stage area in the graphical programming interface based on the real-time instructions.

[0034] The existing graphical programming technology has the following problems: First, the interaction mode is single. The current graphical programming platform mainly relies on the interaction mode of mouse and keyboard, lacking diversified interaction modes. Second, the real-time performance is insufficient. The character actions in the existing platform mainly rely on pre-written instructions and lack real-time interactive control. Finally, the immersion is low. There is a lack of support for user facial expression and gesture recognition, which cannot provide a more immersive experience.

[0035] This application solves the above problems by introducing multiple interaction methods, enhancing real-time control capabilities, and integrating facial expression and gesture recognition technology. Users can not only use traditional mouse and keyboard, but also interact with the programming platform through facial expressions, gestures, etc., providing more diverse interaction modes. At the same time, the platform realizes real-time control of character actions, allowing users to adjust and control character behavior in real time, improving the smoothness and response speed of interaction. In addition, the integrated facial expression and gesture recognition technology enhances the user's sense of immersion, making the programming process more vivid and interesting, and improving the user experience.

[0036] Figure 2 is an architectural diagram of a graphical programming system in a Web browser according to an embodiment of the present application, such as Figure 2 As shown, the system includes a front-end interface 12, a back-end engine 14 and a model library 16. The front-end interface 12 is a graphical programming interface for user operation interaction. The back-end engine 14 is used to process front-end user instructions. The model library contains multiple models.

[0037] The front-end interface 12 is a graphical programming interface for user operation interaction. The graphical programming interface provides an intuitive and user-friendly way for users to create and edit programs by dragging and dropping programming instruction blocks. In addition, the front-end interface 12 is also responsible for real-time feedback of user operations and displaying the status and results of program execution.

[0038] The backend engine 14 is used to process frontend user instructions. Specifically, it is used to receive programming instructions from the frontend interface 12, parse these instructions, and convert them into executable code or instruction sets. The backend engine 14 also includes a runtime environment in which user-defined programs can be executed and data and events generated during program execution can be processed. In addition, the backend engine 14 can also interact with external systems and services to achieve more complex functions and tasks.

[0039] The model library 16 contains a variety of models that can be used for different programming tasks and application scenarios. Each model in the model library 16 is pre-trained and optimized and can be directly applied to the user's programming project. These models can include machine learning models, data analysis models, simulation models, etc. Users can select appropriate models from the model library 16 and integrate them into their own programs to achieve specific functions and goals. Users can also generate and train models through a graphical programming interface and store the trained models in the model library 16. In addition, the model library 16 also supports model updates and expansions. Users can add new models or update existing models as needed to maintain the advancement and flexibility of the system.

[0040] Specifically, the front-end interface 12 is a graphical programming interface, for example, Figure 3 As shown, it includes a programming area 32 (also called a console) for editing graphical codes, a stage area 36 for displaying the execution effect of the graphical codes, and a character bar 34 for providing the stage area 36 with stage characters and backgrounds.

[0041] The programming area 32 is the main operation area for users to perform graphical programming, and includes two components. The first is the code area 322, which is used to display and edit graphical codes. Users can create and modify program logic in this area by dragging and connecting programming instruction blocks. The second is the programming instruction building block box 324, which provides various programming instruction blocks. Users can drag these blocks into the code area 322 to build a program. The building block box contains multiple categories of building blocks, such as the camera building block box 3242, which covers various programming needs from basic operations to advanced functions.

[0042] The stage area 36 is used to display the execution effect of the graphical code. The user can directly observe the program running results by clicking the start button of the stage area 36. Whether it is animation, interaction or data display, the stage area 36 provides a visual platform so that users can instantly see the actual effect of the code they wrote. This is very important for debugging and improving the program, because users can find and correct errors by observing the output effect.

[0043] Figure 4 Click on the display interface of the camera building block box, such as Figure 4 As shown, by clicking the camera building block box 3242 in the programming instruction building block box 324, a plurality of programming instruction building blocks 32422 related to the camera will appear, for example, programming instruction building blocks related to the properties of the camera, and programming instruction building blocks related to the recognition of images collected by the camera. These programming instruction building blocks provide a wealth of functions, and users can use these building blocks to easily control the camera and process images.

[0044] The programming instruction blocks related to the properties of the camera include basic camera operation instructions. For example, users can use the "Turn on / off camera" programming instruction block to control the on / off state of the camera. In addition, there is a "Set opacity" programming instruction block, through which users can adjust the transparency of the camera image to achieve image overlay or fade-in and fade-out effects.

[0045] Blocks related to the recognition of images captured by the camera provide more advanced image processing functions. These blocks can be used to recognize and process various target objects from the camera image. For example, blocks related to faces include multiple specific functions, such as "recognize face", "touch face", "move to face", "face facing", "face coordinates", "face size, width and height", "face tilt angle" and "face tilt left / right". These blocks enable users to control and track facial features in detail, thus realizing applications such as face detection and expression recognition. In addition, there are blocks related to gestures of target objects, which can recognize the gestures of people in the camera image. For example, users can use the "gesture is scissors" block to detect a specific gesture action.

[0046] When users drag these programming instruction blocks to the code area, some blocks will automatically expand to more detailed options. Figure 5 As shown in the figure, when the user drags the programming instruction block of "gesture is scissors" to the code area, the block will be expanded to include multiple gesture options such as "gesture is scissors", "gesture is rock", "gesture is cloth", and "gesture is like". After these programming instruction blocks are dragged to the code area, they can be combined with other blocks to form complex program logic. For example, a user can create a program that performs a specific action or triggers a specific event when a specific gesture is detected. This graphical programming method not only simplifies the programming process, but also greatly lowers the threshold of programming, allowing more people to easily get started and experience the fun of programming.

[0047] The present application embodiment provides another graphical programming method based on image recognition, which is applied to the graphical programming system as described above. Figure 6 As shown, the method comprises the following steps:

[0048] Step S602, performing event binding during graphical code programming.

[0049] When the system is initialized, it registers all possible image recognition events, such as "face recognized", "smile recognized", "hand gesture recognized as scissors", etc. Each event has a unique identifier.

[0050] The user starts the graphical programming platform in the browser or on the APP, and creates various instructions by dragging the programming instruction blocks to perform graphical programming. In the graphical programming interface, the user creates program logic by dragging the programming instruction blocks. For example, the user can drag the "When the gesture is recognized as scissors" block to the code area and add specific instructions below it, such as "rotate 30 degrees". Users can combine multiple blocks together to form a complex logic flow.

[0051] When the user drags a camera-related block to the code area, the system will automatically bind the block to the corresponding event. For example, when the user drags the "When the recognized gesture is scissors" block, the system will bind this block to the "Gesture Recognition Event-Scissors".

[0052] The specific implementation of event binding is completed through event listeners. Whenever the system recognizes an image event, the event listener will check whether a corresponding building block has been dragged to the code area. If so, it will trigger the instructions in the building block.

[0053] Step S604, collecting images.

[0054] The system calls the browser's WebRTC interface, and the camera starts capturing video. WebRTC technology allows the browser to directly obtain the video stream from the camera and transmit it to the platform's image processing module in real time. The low latency feature of WebRTC ensures the immediacy of the video stream, allowing the user's facial expressions and gestures to be captured in real time.

[0055] The captured video stream will be preprocessed to meet the needs of subsequent image recognition. The preprocessing step includes operations such as adjusting the video resolution and controlling the frame rate. These operations ensure that the data quality transmitted to the image recognition module is optimal, providing a guarantee for accurate image recognition.

[0056] Real-time video capture achieved through WebRTC technology not only ensures the immediacy and stability of image input, but also simplifies the user's operation process. Users only need to authorize the camera to use the system, and the system can automatically handle the capture and transmission of video streams without manual configuration, which improves the user experience.

[0057] Step S606, identifying the image.

[0058] The system analyzes these captured images in real time to recognize the user's facial expressions and gestures and generate corresponding events. Figure 7 As shown, the following steps are included:

[0059] Step S6062: image preprocessing.

[0060] Extract single-frame images from a continuous video stream. A certain number of image frames are extracted every second to ensure that the system can respond to changes in the user's expressions and gestures in a timely manner. The extracted image frames are standardized, including resizing and grayscale processing, to meet the input requirements of the model. This step can improve the efficiency and accuracy of image processing.

[0061] First, perform image normalization. Each pixel value of the image is converted and scaled to the range of [0,1] to reduce the numerical differences in the original image caused by different lighting conditions or camera equipment. This standardization process ensures that the image data has a consistent scale and distribution when it is input into the model, which significantly speeds up the training and convergence of the model and improves the stability and prediction effect of the model.

[0062] Next, perform data augmentation operations to increase the generalization ability of the model and prevent overfitting. For example: randomly rotate the image, with the angle varying within a preset range; randomly scale the image size to adjust the overall perspective of the image; randomly translate the image horizontally or vertically to change the position of the target object in the image; randomly cut part of the image to remove the background or unimportant parts; randomly flip the image, both horizontally and vertically; randomly adjust the brightness, contrast, and saturation of the image to simulate different shooting conditions and lighting effects. Through these operations, a variety of training samples can be generated to improve the model's adaptability to different environments and scenes.

[0063] After data enhancement, image denoising is performed to reduce the image quality degradation caused by sensor noise or compression algorithms. Image denoising is an image processing technique whose main purpose is to remove or reduce noise from noise-affected images while retaining the original features and details of the image as much as possible. In the process of image acquisition or transmission, some random noise signals are usually introduced, which will have an adverse effect on the quality and usability of the image. Noise types include Gaussian noise, salt and pepper noise, Poisson noise, and multiplicative noise. Different noise types often require different denoising methods to be used. The main methods of image denoising can be divided into several categories, including spatial domain filtering methods, frequency domain filtering methods, adaptive filtering methods, and deep learning-based methods. Spatial domain filtering methods such as mean filtering, median filtering, and Gaussian filtering reduce noise by smoothing pixels in the spatial domain of the image. Frequency domain filtering methods such as Fourier transform and wavelet transform remove high-frequency noise by converting the image to the frequency domain. Adaptive filtering methods, such as Wiener filtering and non-local means method, perform adaptive filtering based on the local statistical characteristics or global similarity of the image, which can more effectively preserve the details and edges of the image.

[0064] Specifically, first, it is necessary to generate a Gaussian kernel for smoothing the image, and normalize the kernel to ensure that the sum of the kernel is 1, thereby maintaining the overall brightness of the image. Then, the generated Gaussian kernel is used to calculate the offset value of each pixel in the image in combination with the spatial information, horizontal and vertical gradients of each pixel. The offset value is used to describe the grayscale changes and texture details of the image in a local area. In addition, in some embodiments, the noise intensity parameter can also be considered as an adjustment factor to control the sensitivity of the denoising process. Specifically, the width of the Gaussian distribution can be determined based on the standard deviation, and the center of the Gaussian kernel can be constructed based on the width of the Gaussian distribution, the coordinates of the pixels in the image relative to the center of the kernel, and the size of the Gaussian kernel. Then, denoising is performed based on the pixel value of the original image at position (i, j), the offset of the horizontal coordinate position and the vertical coordinate position relative to the center of the Gaussian kernel, the horizontal gradient of the image at each pixel position, the vertical gradient of the image at each pixel position, the gradient weight, and the noise intensity parameter used to control the sensitivity to the difference in pixel values ​​of the image during the denoising process, wherein the noise intensity parameter is obtained based on the noise intensity, the pixel value of the image at position (i, j), and the pixel value of the image at position (i, j) after the offset. The denoising method effectively retains the edge and detail information of the image by utilizing the pixel value, the offset relative to the center of the Gaussian kernel, the horizontal and vertical gradients, the gradient weight, and the adaptive noise intensity parameter based on the pixel value and the noise intensity, thereby improving the denoising accuracy and reducing the artifact effect. It has high robustness to different types of noise and is suitable for use in applications requiring high-quality denoising, such as high-resolution images and real-time video processing.

[0065] During the entire image preprocessing process, different steps can be iterated multiple times. For example, for the denoising and data enhancement parts, the parameters can be adjusted for repeated processing to further optimize the quality and diversity of the image, ensure that the final image preprocessing results are optimal, and provide higher quality input data for the subsequent training and application of deep learning models.

[0066] Step S6064, model loading.

[0067] The system first loads two trained deep learning models: one for face recognition and the other for gesture recognition. These models are based on convolutional neural network (CNN architecture, such as multi-task learning network or deep model combined with attention mechanism. The face recognition model can effectively detect the face area in the image and identify key facial feature points (such as eyes, nose, mouth, face contour, etc.), while the gesture recognition model focuses on identifying and classifying the user's hand posture type (such as open, clenched fist, heart shape, etc.).

[0068] Step S6066, feature extraction.

[0069] The face model is used to detect and extract features of the face in each frame of the image. The extracted features include facial expressions (such as smile, frown, etc.) and the position and angle of facial feature points. The gesture model is used to detect and extract features of the gesture in each frame of the image. The extracted features include gesture type (such as scissors, rock, cloth, etc.) and the position and angle of the hand.

[0070] Step S6068, event generation.

[0071] According to the results of facial feature extraction, the corresponding facial expression events are generated. For example, when a user smiles, a "smile recognized" event is generated. According to the results of gesture feature extraction, the corresponding gesture events are generated. For example, when a user's gesture is detected as scissors, a "scissors recognized" event is generated.

[0072] Through the above steps, the system can efficiently and accurately recognize the user's facial expressions and gestures and generate corresponding events, providing a reliable basis for the subsequent execution of programming instructions.

[0073] Step S608, event execution.

[0074] Associate the recognized events with specific programming instructions. When the system detects an image recognition event at runtime (for example, the user makes a scissors gesture), the event listener captures the event and looks for a programming block bound to the event. If the corresponding block is found, the system immediately executes the instruction under the block. For example, if the user drags the "When the recognized gesture is scissors" block and adds a "Rotate 30 degrees" instruction below it, when the system recognizes the scissors gesture, the character will immediately rotate 30 degrees.

[0075] This application uses a graphical programming interface, and users can intuitively associate complex image recognition functions with programming instructions without writing code, which greatly reduces the difficulty of programming. This intuitive operation method enables more users to participate in and enjoy the fun of programming. Even users who have no programming foundation can create interactive programs by dragging and dropping building blocks. In addition, users can drag and combine different building blocks as needed to form various complex logical processes. The system can also add more recognition events and corresponding building blocks according to actual needs to expand programming functions. Through the event listener, the system can respond to image recognition events in real time and immediately execute the corresponding programming instructions, which enhances the interactivity and response speed of the system. Through the above implementation method, the system not only simplifies the operation of complex image recognition functions, but also provides an intuitive and flexible programming experience, allowing more users to participate in programming creation and enjoy the fun of graphical programming.

[0076] The present application also provides a graphical programming device based on image recognition, such as Figure 8As shown, it includes: an image acquisition module 82, which is configured to collect images in real time through a camera in response to detecting that the graphical code in the programming area in the graphical programming interface includes programming instruction blocks related to the camera; a recognition module 84, which is configured to recognize the collected image based on the programming instruction blocks related to the camera to obtain an image recognition result, and generate real-time instructions based on the image recognition result; a control module 86, which is configured to control the actions of the stage characters in the stage area in the graphical programming interface based on the real-time instructions.

[0077] It should be noted that: the graphical programming device based on image recognition provided in the above embodiment is only illustrated by the division of the above functional modules. In actual application, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the graphical programming device based on image recognition provided in the above embodiment and the graphical programming method embodiment based on image recognition belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0078] Fig. 9 FIG. 1 shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present disclosure. It should be noted that: Fig. 9 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0079] like Fig. 9 As shown, the electronic device includes a central processing unit (CPU) 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage part 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for system operation are also stored. The CPU 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0080] The following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, etc.; an output section 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, a modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed, so that a computer program read therefrom is installed into the storage section 1008 as needed.

[0081] In particular, according to an embodiment of the present disclosure, the process described below with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1009, and / or installed from the removable medium 1011. When the computer program is executed by the central processing unit (CPU) 1001, various functions defined in the method and apparatus of the present application are executed. In some embodiments, the electronic device may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0082] It should be noted that the computer-readable medium shown in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0083] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. An AI graphical programming method based on image recognition, characterized in that: include: In response to detecting that the graphical code of the programming area in the graphical programming interface includes programming instruction blocks associated with the camera, real-time image acquisition via the camera; Based on the programming instruction building blocks related to the camera, the collected image is identified to obtain an image recognition result, and a real-time instruction is generated based on the image recognition result; Based on the real-time instructions, the actions of the stage characters in the stage area in the graphical programming interface are controlled.

2. The method according to claim 1, characterized in that The graphical programming interface includes the programming area for editing graphical code and the stage area for displaying the execution effect of the graphical code, wherein the programming area includes a code area for displaying the graphical code and a programming instruction building block box for providing programming instruction building blocks for the code area, wherein the programming instruction building block box includes a camera building block box, and the camera building block box is used to provide the programming instruction blocks related to the camera.

3. The method according to claim 1, characterized in that The camera-related programming instruction building blocks include at least one of the following: Programming instruction blocks related to the properties of the camera, wherein the programming instruction blocks related to the properties of the camera include programming instruction blocks for controlling the camera switch and programming instruction blocks for controlling the transparency of the camera imaging; Programming instruction blocks related to recognition of images captured by the camera, wherein the programming instruction blocks related to image recognition include at least one of the following: programming instruction blocks related to the face of a target object recognized from the image and programming instruction blocks related to a gesture of the target object.

4. The method according to claim 3, characterized in that Based on the programming instruction building blocks related to the camera, the collected image is identified to obtain an image recognition result, including: In the case where the programming instruction block associated with the camera is a programming instruction block associated with the face of the target object, identifying the face of the target object, and determining whether the face of the target object meets the condition indicated by the programming instruction block associated with the face of the target object, so as to obtain the image recognition result of the face; In the case where the programming instruction block related to the camera is a programming instruction block related to the gesture of the target object, the gesture of the target object is identified, and it is determined whether the gesture of the target object meets the conditions indicated by the programming instruction block related to the gesture of the target object, so as to obtain the image recognition result of the gesture.

5. The method according to claim 4, characterized in that Generating a real-time instruction based on the image recognition result includes: In the case where the camera-related programming instruction block is a programming instruction block related to the face of the target object, generating the real-time instruction based on the image recognition result of the face and the graphical code of the programming area; In a case where the camera-related programming instruction block is a programming instruction block related to a gesture of the target object, the real-time instruction is generated based on an image recognition result of the gesture and a graphical code of the programming area.

6. The method according to claim 2, characterized in that Before responding to detecting that the graphical code in the programming area in the graphical programming interface includes programming instruction blocks related to the camera, the method also includes: responding to detecting that one or more programming instruction blocks in the programming instruction block box are dragged from the programming instruction block box to the code area, generating the graphical code corresponding to the one or more programming instruction blocks in the code area.

7. An AI graphical programming device based on image recognition, characterized in that: include: An image acquisition module configured to acquire an image in real time through a camera in response to detecting that a graphical code in a programming area in a graphical programming interface includes a programming instruction building block associated with a camera; A recognition module, configured to recognize the collected image based on the programming instruction building block related to the camera to obtain an image recognition result, and generate a real-time instruction based on the image recognition result; The control module is configured to control the actions of the stage characters in the stage area in the graphical programming interface based on the real-time instructions.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 6.

9. A computer device, characterized in that: include: Memory and processor, The memory stores a computer program; The processor is used to execute the computer program stored in the memory, and when the computer program is run, the processor is enabled to execute the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images