Task program execution method, device, electronic device and storage medium

Abstract

The present disclosure provides a task program execution method, device, electronic device, and storage medium. The method comprises: executing the task program and determining the current execution progress of the task program; obtaining a distorted image of the current execution progress, correcting the distorted image according to image correction parameters of the task program, and obtaining a corrected image of the current execution progress; comparing image detection information of the corrected image with image reference information of the current execution progress; and if the image detection information matches the image reference information, continuing to execute the task program until the task program is completed. Thus, the present disclosure can accurately control the execution progress of the task program based on a simple image detection algorithm, reducing system operating load and hardware configuration requirements.

Description

Technical Field

[0001] The present invention relates to the field of image detection technology, and in particular to a task program execution method, device, electronic device, and storage medium. Background Art

[0002] With the continuous deepening of quality education, quality cultivation and ability training have become important contents of education. Especially in early childhood, the development of good thinking habits plays a vital role in future development. Among them, educational games can improve students' learning efficiency and enthusiasm by transforming knowledge points into interesting games. Therefore, in current early childhood education, educational games play an important role.

[0003] During the execution of educational games, it is also very important to be able to give children timely game feedback. Game feedback refers to the ability to provide relevant execution feedback when children play correctly or incorrectly, get correct or incorrect results, or are unable to pass a level for a long time, so as to guide children to successfully complete various levels in the task program. The implementation of game feedback technology relies on the deep involvement of intelligent algorithms and discrimination strategies, which will increase the game operation load and put higher requirements on hardware configuration.

[0004] In view of this, there is an urgent need for a technical solution that can reduce system operation load and provide accurate mission execution guidance. Summary of the Invention

[0005] In view of this, embodiments of the present disclosure provide a task program execution method, apparatus, electronic device, and storage medium to at least partially solve the above-mentioned problems.

[0006] According to the first aspect of the present disclosure, a task program execution method is provided, including: executing the task program, determining the current execution process of the task program; obtaining a distorted image of the current execution process, performing correction on the distorted image of the current execution process according to the image correction parameters of the task program, and obtaining a corrected image of the current execution process; performing comparison based on image detection information of the corrected image and image reference information of the current execution process, and if the image detection information matches the image reference information, continuing the steps of executing the task program until the execution of the task program is completed.

[0007] According to a second aspect of the present disclosure, a task program execution device is provided, comprising: an execution unit, which is used to execute the task program and determine the current execution process of the task program; an acquisition unit, which is connected to the execution device and is used to acquire a distorted image of the current execution process; the execution device is also used to: perform correction on the distorted image of the current execution process according to the image correction parameters of the task program to obtain a corrected image of the current execution process; perform comparison based on image detection information of the corrected image and image reference information of the current execution process, and if the image detection information is consistent with the image reference information, continue to execute the task program until the execution of the task program is completed.

[0008] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing a program, wherein the program comprises instructions that, when executed by the processor, cause the processor to execute the task program execution method described in the first aspect above.

[0009] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the task program execution method as described in the first aspect.

[0010] In summary, the task program execution scheme provided by various aspects of this disclosure obtains a corrected image of the current task program execution process, corrects the distorted image using given image correction parameters, and controls the execution progress of the task program based on the detection results of the corrected image. This allows the present disclosure to accurately control the execution progress of the task program based on a simple image detection algorithm, reducing system operating load and hardware configuration requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Further details, features and advantages of the present disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0012] Figure 1 Schematic diagram of a system architecture applicable to the task program execution method disclosed herein.

[0013] Figure 2A and Figure 2B This is a diagram showing an application example of a placement game suitable for the task program execution method disclosed herein.

[0014] Figure 3 This is a processing flow chart of a task program execution method according to an exemplary embodiment of the present disclosure.

[0015] Figures 4A to 4C This is a schematic diagram of an application of a task program execution method according to an exemplary embodiment of the present disclosure.

[0016] Figure 5 This is a processing flow chart of a task program execution method according to another exemplary embodiment of the present disclosure.

[0017] Figures 6A to 6B A schematic diagram illustrating an application of a task program execution method according to another exemplary embodiment of the present disclosure.

[0018] Figure 7 This is a processing flow chart of a task program execution method according to another exemplary embodiment of the present disclosure.

[0019] Figure 8 This is a processing flow chart of a task program execution method according to another exemplary embodiment of the present disclosure.

[0020] Figure 9 This is a processing flow chart of a task program execution method according to another exemplary embodiment of the present disclosure.

[0021] Figure 10 This is a processing flow chart of a task program execution method according to another exemplary embodiment of the present disclosure.

[0022] Figure 11 FIG. 4 is a structural block diagram of a task program execution device according to an exemplary embodiment of the present disclosure.

[0023] Figure 12 FIG. 1 is a schematic diagram of the architecture of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0025] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0026] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0027] It should be noted that the modifiers "one" and "plurality" mentioned in this disclosure are illustrative and non-restrictive. Those skilled in the art will understand that unless the context clearly indicates otherwise, they should be understood as "one or more." The names of the messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0028] As described above in the background technology section, in current puzzle games, the implementation of game feedback technology relies on the deep involvement of intelligent algorithms and discrimination strategies, and intelligent algorithms and discrimination strategies are mostly implemented through deep learning models. Since deep learning models have high requirements for hardware configuration, and the training and subsequent deployment of deep learning models require a lot of time and financial costs, in view of this, the present disclosure proposes a task program execution solution that can reduce the system operating load and hardware configuration requirements.

[0029] In some embodiments, the task program of the present disclosure includes a game program of a placement game. For ease of understanding, the game background and gameplay of the placement game of the present disclosure are introduced below.

[0030] Figure 1 The following is a schematic diagram showing the system architecture of a placement game suitable for the task program execution method disclosed in the present invention. Figure 1 As shown, the system 100 may include an electronic device 102 and a game board 104. The electronic device 102 may be, for example, a smart desk lamp, a smart tablet, or a computer, and is configured to host and execute a game program for a placement game. The game board 104 is drawn with a plurality of squares arranged in a matrix.

[0031] In some embodiments, the electronic device 102 is provided with a camera 1022 and a display screen 1024, wherein the display screen 1024 is used to display the operation prompts of each game level in the game program, and the user can place the corresponding game accessories on the game base 104 according to the operation prompts on the display screen 1024; the camera 1022 can take pictures of the game accessories in the game base 102 to detect whether the placement of each game accessory on the game base 102 is consistent with the operation prompts of the currently executed game level.

[0032] Specifically, placement games have a background story or theme, such as inviting small animals to stay in a hotel. They primarily cultivate the user's understanding of shapes, directions, and distances. First, the game program displays operating instructions for each game level on the display screen 1024. The user follows these instructions to place designated game accessories, such as game cards, geometric blocks, and chess pieces, within designated squares on the game board 104. In this embodiment, cards are used as an example.

[0033] For example, the display screen 1024 may be used to display an operation prompt 200 such as "Please move the lion into the room in the 3rd row and 4th column" or "Please move the lion up two steps." The user may perform related operations on the game base 104 according to the operation prompt 200 on the display screen 1024. For example, when the operation prompt 200 "Please move the lion into the room in the 3rd row and 4th column" is displayed on the display screen 1024, the user may place the "lion card 105a" in the square in the 3rd row and 4th column of the game base 104 (refer to Figure 2A ); For another example, when the operation prompt 200 of "Please move the lion two steps up" is displayed on the display screen 1024, the user can move the "lion card 105a" from the square in the 3rd row and 4th column of the game base 104 to the square in the 1st row and 4th column of the game base 104 (refer to Figures 2A to 2B ).

[0034] After the user performs relevant operations on the game base 104 according to the operation prompt 200 on the display screen 1024, the camera 1022 can shoot the game base 104 to detect whether the placement positions of the game accessories on the game base 104 are consistent with the operation prompt 200 on the display screen 1024, thereby determining whether the current game level of the game program has been passed. Through this placement game program, preschool children can be helped to correctly understand concepts such as direction (up, down, left, right, east, south, west, north, etc.), rows, columns, numbers, geometric shapes, and distance.

[0035] Specific embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0036] Figure 3This is a process flow chart of a task program execution method according to an exemplary embodiment of the present disclosure. As shown in the figure, this embodiment mainly includes the following steps:

[0037] Step S302: execute the task program and determine the current execution process of the task program.

[0038] In this embodiment, the task program (such as a game program for a placement game) can be loaded into the electronic device 102 (refer to Figure 1 ), the electronic device 102 may execute the task program according to the triggering instruction of the task program.

[0039] Optionally, the task program may include a task preparation process and at least one task operation process following the task preparation process.

[0040] Optionally, an operation prompt of the currently executed process may be output (eg, via the display screen 1024 on the electronic device 102 ).

[0041] Step S304: obtaining a distorted image of the currently executed process, and performing correction on the distorted image of the currently executed process according to the image correction parameters of the task program to obtain a corrected image of the currently executed process.

[0042] For example, the distorted image of the current execution process can be collected (refer to Figure 4A 、 Figure 4B ), and use the image correction parameters of the task program to perform the correction processing of the distorted image to obtain the corrected image of the current execution process (reference Figure 4C ).

[0043] In this embodiment, the distorted image 400 including the basic accessories and / or optional accessories of the task program can be obtained according to the current execution process of the task program (refer to Figure 4A 、 Figure 4B ).

[0044] In this embodiment, perspective transformation technology can be used to determine the image correction parameters for a task program before executing it. This allows a single detection model to perform image correction and detection based on the predetermined image correction parameters during the task program, eliminating the need for a deep learning model to perform image correction. This reduces hardware requirements and system computational overhead, speeding up the task program's execution.

[0045] Step S306 , comparing the image detection information of the corrected image with the image reference information of the current execution process to see if they are consistent. If so, proceed to step S308 ; otherwise, proceed to step S304 .

[0046] Optionally, detection may be performed on the corrected image to obtain image detection information of the corrected image, and comparison may be performed based on the image detection information of the corrected image and the image reference information of the current execution process to select execution of step S308 or step S304 based on the comparison result.

[0047] Optionally, when the current execution process of the task program is the task preparation process, the image reference information of the task preparation process may include: a reference viewing range of the task program and basic accessory reference information of the task program.

[0048] In this embodiment, the basic component reference information includes a reference category of the basic component of the task program and a reference positioning position of the basic component relative to an actual viewing range of the corrected image.

[0049] Optionally, in a case where the current execution process of the task program is a task operation process, the image reference information of the task operation process may include optional accessory reference information.

[0050] In this embodiment, the optional accessory reference information includes a reference category of the optional accessory of the task program and a reference positioning position of the optional accessory relative to the basic accessory.

[0051] Optionally, the basic accessories of the task program include a base plate (eg, a game base plate), and the base plate has a plurality of blocks (eg, grids) of the same size and arranged in a matrix form.

[0052] Optionally, optional accessories of the task program include game cards, game geometric blocks, game pieces, etc. that can be placed in various blocks on the base plate.

[0053] Optionally, the operation prompt of the currently executed process is consistent with the image reference information of the currently executed process.

[0054] For example, reference Figure 2A and Figure 2B , when the "Lion Card 105a" is located in the square in the 3rd row and 4th column of the game base (basic accessory), if the current execution process (task operation process) gives the operation prompt 202 of "Please move the lion two steps up", the reference category of the optional accessory in the optional accessory reference information of the current execution process (task operation process) should be "Lion", and the reference positioning position of the optional accessory relative to the basic accessory should be "the square in the 1st row and 4th column of the game base".

[0055] Specifically, if the image detection information of the currently executing process is consistent with the image reference information, indicating that the currently executing process is completed (for example, the current game level of the game program is passed), then step S308 is performed; conversely, if the image detection information of the currently executing process is inconsistent with the image reference information, indicating that the current executing process is not completed (for example, the current game level of the game program is not passed), then step S304 is performed to re-acquire the distorted image of the currently executing process and perform detection.

[0056] Step S308, determine whether the task program is completed. If so, end this process. If not, execute step S302.

[0057] Specifically, it can be determined whether the task preparation program and each task operation program in the task program have been executed. If so, the execution of the task program is terminated. If not, step S302 is executed to continue executing the next process of the task program.

[0058] In summary, the task program execution method provided by this embodiment obtains the distorted image of the current execution process, corrects the distorted image based on given image correction parameters, and detects whether the corrected image is consistent with the image reference information of the current execution process, thereby controlling the execution progress of the task program. In this way, the present application can achieve accurate control of the execution progress of the task program based on a simple image detection algorithm. Specifically, this embodiment utilizes a simple image detection algorithm instead of a traditional deep learning model to perform image correction and detection processing, which not only reduces hardware configuration requirements and reduces system operating load to increase the execution speed of the task program, but also avoids the training of deep learning models and subsequent deployment operations, which can save a lot of manpower and time costs.

[0059] Figure 5 This is a flowchart of a method for executing a task program according to another exemplary embodiment of the present disclosure. This embodiment illustrates a technical solution for determining image correction parameters for a task program, which mainly includes the following steps:

[0060] Step S502: Acquire a reference graph of the task program to obtain a distorted image including a distorted graph.

[0061] For example, a white paper 110 (reference Figure 6A ), and use the camera 1022 of the electronic device 102 to capture the image of the white paper 110, and obtain the distorted image 400 with the distorted pattern 112 (refer to Figure 6B ).

[0062] Optionally, the reference graphic 112 ′ of the task program may be a rectangle or a square.

[0063] For example, when the reference pattern 112 ′ is a square, the distortion pattern 112 of the reference pattern 112 ′ may be a trapezoid.

[0064] Step S504: Detect each vertex of the distorted pattern to obtain a plurality of distorted coordinates of the distorted pattern.

[0065] Specifically, refer to Figure 6B , the corner point positions of the distorted pattern 112 of the distorted image 400 can be detected to obtain the distorted coordinates (x1, y1), (x2, y2), (x3, y3), (x4, y4) of the distorted pattern 112.

[0066] Step S506 : determining a plurality of reference coordinates and a plurality of coordinates to be calibrated among the plurality of distorted coordinates according to the shooting posture of the distorted image.

[0067] In this embodiment, the reference edge of the distorted image can be determined according to the shooting posture of the distorted image, and the two distorted coordinates of the two corner points of the reference edge are determined as reference coordinates, and the distorted coordinates that are not determined as reference coordinates are determined as coordinates to be calibrated.

[0068] Optionally, when the longitudinal direction of the reference pattern 112' (for example, along Figure 6A or Figure 6B When collecting the reference pattern 112' along the Y-axis direction), one of the two side lines of the reference pattern 112' along the horizontal direction (for example, the upper and lower horizontal sides of the reference pattern 112' along the X-axis direction) can be determined as the reference side line; when collecting the reference pattern 112' along the horizontal direction (for example, the upper and lower horizontal sides of the reference pattern 112' along the X-axis direction) along the Y-axis direction, one of the two side lines of the reference pattern 112' along the horizontal direction (for example, the upper and lower horizontal sides of the reference pattern 112' along the X-axis direction) can be determined as the reference side line. Figure 6A or Figure 6B When collecting the reference pattern 112' along the X-axis direction), one of the two side lines of the reference pattern 112' along the longitudinal direction (for example, the left and right vertical sides of the reference pattern 112' along the Y-axis direction) can be determined as the reference side line.

[0069] exist Figure 6B In the example shown, based on the shooting posture of the distorted image 400, the edge 113 of the reference figure 112' closest to the camera 1022 can be determined as the reference edge, and the two distorted coordinates (x1, y1) and (x2, y2) of the two corner points of the reference edge 113 can be determined as reference coordinates, and the two distorted coordinates (x3, y3) and (x4, y4) that have not been determined as reference coordinates can be determined as coordinates to be calibrated.

[0070] Step S508 : Calibrate the multiple coordinates to be calibrated according to the reference pattern and the multiple reference coordinates to obtain multiple calibration coordinates of the multiple coordinates to be calibrated.

[0071] For example, when the reference figure 112' is a square, calibration can be performed on the two coordinates to be calibrated (x3, y3) and (x4, y4) according to the two reference coordinates (x1, x1) and (x2, y2), respectively, to obtain the calibration coordinates (x2, y2+(x2-x1)) corresponding to the coordinate to be calibrated (x3, y3), and the calibration coordinates (x1, y1+(x2-x1)) corresponding to the coordinate to be calibrated (x4, y4).

[0072] In step S510 , image correction parameters of the task program are obtained according to a first coordinate set consisting of the distorted coordinates and a second coordinate set consisting of the reference coordinates and the calibration coordinates.

[0073] Optionally, the distorted coordinates may be arranged in a clockwise or counterclockwise order to obtain a first coordinate set. Figure 6B In the example shown, the distorted coordinates can be arranged in a clockwise order to obtain a first coordinate set, which is recorded as {(x1, y1), (x2, y2), (x3, y3), (x4, y4)}; and the reference coordinates and the calibration coordinates are arranged in the same order as the first coordinate set to obtain a second coordinate set (in this embodiment, the reference coordinates are (x1, y1) and (x2, y2), and the remaining two coordinates are determined by their respective calibration coordinates based on the reference coordinates, thereby obtaining the second coordinate set), which is recorded as {(x1, y1), (x2, y2), (x2, y2+(x2-x1)), (x1, y1+(x2-x1))}.

[0074] In this embodiment, the perspective transformation matrix of the task program can be obtained based on the first coordinate set and the second coordinate set by calling a function for calculating the perspective transformation matrix in a computer vision library (eg, opencv).

[0075] It should be noted that when the image correction parameter of the task program is a perspective transformation matrix, when executing the above step S304, it is also possible to call the function for calculating the perspective transformation matrix in the computer vision library (such as opencv) to perform correction on the distorted image of the current execution process based on the perspective transformation matrix of the task program to obtain the corrected image of the current execution process.

[0076] In summary, the task program execution method of this embodiment can determine the reference coordinates and coordinates to be calibrated in each distorted coordinate system based on the distorted image's shooting posture. Using the principle of image perspective transformation, the image correction parameters of the task program are determined based on the reference image and reference coordinates. This allows this embodiment to determine the image correction parameters corresponding to different distorted image shooting postures, thereby increasing the flexibility of the image correction process.

[0077] Figure 7This is a process flow chart of a task program execution method according to another exemplary embodiment of the present disclosure. This embodiment shows a specific implementation scheme of step S306 when the current execution process of the task program is the task preparation process. As shown in the figure, it mainly includes the following steps:

[0078] Step S702 : detecting the corrected image of the task preparation program to determine the actual viewing range of the task preparation program.

[0079] refer to Figure 1 In the example shown, when the electronic device 102 is a tablet computer, the electronic device 102 can be fixed using a bracket 106, and a reflector 108 can be set on the electronic device 102 to use the reflector 108 to provide a camera 1022 on the electronic device 102 to capture images of the game base 104 placed on the desktop 101.

[0080] exist Figure 1 In the illustrated embodiment, the bracket 106 and / or the reflector 108 may be adjusted to adjust and determine the actual viewing range of the mission preparation procedure.

[0081] It should be noted that the actual viewing range of the task preparation process can also be adjusted and determined through other methods. For example, if the camera is movable relative to the electronic device (e.g., the camera can rotate relative to the electronic device), the camera's viewing range can be adjusted by adjusting the camera's position relative to the electronic device (e.g., adjusting the camera's rotation angle relative to the electronic device).

[0082] In this embodiment, the actual viewing range of the task preparation process can be determined by detecting the captured object in the rectified image of the task preparation process.

[0083] In step S704 , it is determined whether the actual framing range matches the reference framing range. If so, the process proceeds to step S706 . If not, the process proceeds to step S702 .

[0084] Optionally, when it is detected that the corrected image of the task preparation program includes a designated object, a determination result that the actual framing range is consistent with the reference framing range may be obtained.

[0085] In this embodiment, when it is detected that the two stent marks 1062 of the stent 106 are included in the corrected image of the task preparation process, a determination result is obtained that the actual framing range of the task preparation process is consistent with the reference framing range.

[0086] Optionally, when the judgment result that the actual viewing range of the task preparation program does not match the reference viewing range is obtained, an operation prompt for adjusting the actual viewing range can be output, for example, Figure 1The display screen 2014 shows an operation prompt “Please adjust the bracket 106 and / or the reflector 108”, and the process returns to step S702.

[0087] Step S706 , detecting the actual category of the basic accessories in the corrected image and the actual positioning position of the basic accessories relative to the actual viewing range to obtain basic accessory detection information.

[0088] In this embodiment, the basic component may include a base plate 104 having a plurality of blocks of the same size arranged in a matrix form (see Figures 4A to 4C ).

[0089] For example, the actual type of the baseboard 106 (e.g., Figure 4C 6*6 base plate shown).

[0090] For example, the actual position of the base accessory relative to the actual viewing range may be determined by detecting the distance between the bounding box 107 of the base plate 104 and the edge of the rectified image.

[0091] Step S708 , determining whether the basic accessory detection information is consistent with the basic accessory reference information; if so, executing step 302 ; otherwise, executing step S706 .

[0092] In this embodiment, the reference category of the basic accessory and the reference positioning position of the basic accessory relative to the actual viewing range can be determined based on the basic accessory reference information of the task preparation process. When it is judged that the reference category of the basic accessory is consistent with the actual category, and the reference positioning position of the basic accessory relative to the actual viewing range is also consistent with the actual positioning position, a judgment result is obtained that the basic accessory detection information is consistent with the basic accessory reference information.

[0093] In this embodiment, if it is determined that the basic component detection information is consistent with the basic component reference information, step S302 is executed to continue executing the task operation procedure following the task preparation procedure.

[0094] In this embodiment, if it is determined that the basic accessory detection information does not match the basic accessory reference information, an operation prompt for adjusting the base plate type and / or adjusting the base plate position may be output, and the process returns to step S706 .

[0095] To sum up, the technical solution of this embodiment has performed correction on the distorted image of the task preparation program. Therefore, there is no need to use key point detection. Only by detecting the distance between the specified marks, basic accessories and image edges in the corrected image, it is possible to determine whether the preparation work of the task program is performed correctly, thereby reducing the system operation load and improving the program operation speed.

[0096] Figure 8 This is a processing flow chart of a task program execution method according to another exemplary embodiment of the present disclosure. This embodiment shows another specific implementation of step S306 when the current execution process of the task program is any task operation process. As shown in the figure, it mainly includes the following steps:

[0097] Step S802 : Detect the actual category of the optional accessories in the corrected image and the actual positioning position of the optional accessories relative to the basic accessories to obtain optional accessory detection information.

[0098] In this embodiment, the actual category of the optional accessory can be determined by detecting the image on the optional accessory and / or the shape of the optional accessory, for example, Figure 2A or Figure 2B The "Lion Card 105a" shown, Figure 4A or Figure 4B The "turtle card 105b", "elephant card 105c" or "triangle block 105d" shown.

[0099] Optionally, you can Figure 9 The steps shown are to detect the actual positioning position of the optional accessories relative to the basic accessories:

[0100] Step S902 : determining a block size according to the baseboard size of the baseboard, the number of rows and the number of columns of the multiple blocks.

[0101] Specifically, the size of the bottom plate 104 may be determined based on the bounding box 107 of the bottom plate 104, and the actual category of the bottom plate 104 (e.g. Figure 4C 6*6 base in the bottom panel), determine the number of rows and columns of blocks included in the bottom panel 106, and calculate the size of each block based on the bottom panel size, the number of rows and columns of each block (for example, in Figure 4C In the example shown, the side length of the block = the side length of the bounding box 107 / 6).

[0102] Step S904 : obtaining a lateral difference value of the optional accessory along the lateral direction and a longitudinal difference value of the optional accessory along the longitudinal direction according to the center coordinate of the optional accessory and any vertex coordinate of the base plate.

[0103] Specifically, the center coordinates of the optional accessories (e.g., the “turtle card 105b,” the “elephant card 105c,” or the “triangle block 105d”) can be determined based on the detection frame results of the optional accessories, and the horizontal difference between the center coordinates of the optional accessories and the upper left corner coordinates of the base plate 104 along the horizontal direction (e.g., the X-axis direction) and the vertical difference along the vertical direction (e.g., the Y-axis direction) can be calculated. For example, Figure 4C In the example shown, the horizontal difference of the “elephant card 105 c ” is 330 pixels, and the vertical difference is 200 pixels.

[0104] Step S906 , obtaining the actual number of rows and columns of the blocks where the optional accessories fall in each block according to the horizontal difference, the vertical difference, and the block size.

[0105] Specifically, the ratios of the horizontal difference and the vertical difference relative to the side length of the block can be calculated to obtain the horizontal ratio and vertical ratio of the optional accessory, and the horizontal ratio and vertical ratio of the optional accessory can be rounded up to obtain the number of rows and columns of the actual block in which the optional accessory falls.

[0106] This rounding-up processing method eliminates the need to set an additional distance threshold, and has a certain degree of compatibility with the placement deviation of optional accessories relative to basic accessories. It can well adapt to the phenomenon that young children's hand movements are not yet flexible, thereby improving the children's gaming experience.

[0107] Step S804, determining whether the optional accessory detection information is consistent with the optional accessory reference information; if so, executing step 302; if not, executing step S802.

[0108] Specifically, the reference category of the optional accessory and the reference positioning position of the optional accessory relative to the basic accessory can be determined based on the optional accessory reference information of the task operation process. When it is judged that the reference category of the optional accessory is consistent with the actual category, and the reference positioning position of the optional accessory relative to the basic accessory is also consistent with the actual positioning position, a judgment result is obtained that the optional accessory detection information is consistent with the optional accessory reference information.

[0109] In this embodiment, if it is determined that the optional accessory detection information is consistent with the optional accessory reference information, step S302 is executed to continue executing the next task operation procedure.

[0110] In this embodiment, if it is determined that the optional accessory detection information does not match the optional accessory reference information, an operation prompt such as an optional accessory category error and / or an optional accessory access location error may be output, and the process returns to step S802.

[0111] In summary, the technical solution of this embodiment avoids the need to individually detect the position of each block in the baseplate or all the block dividing lines within it. Furthermore, since the captured image of the currently executing process has been corrected to a top-down view, there's no need to use key point detection technology to separately detect the four corner points of the baseplate. Furthermore, the technical solution of this embodiment integrates all the information to be detected from all execution processes of a task program (the task preparation process and each task operation process) into a single detection model, reducing the computing power burden on the system.

[0112] Figure 10 This is a processing flow chart of a task program execution method of another exemplary embodiment of the present application. As shown in the figure, this embodiment mainly includes the following steps:

[0113] Step S1002 : obtaining a plurality of task-corrected images of the task operation process based on a preset time interval.

[0114] Specifically, when the current execution process of the detection task program is any task operation process, the task correction images of each frame of the task operation process can be obtained intermittently according to a preset time interval.

[0115] Step S1004 , according to each acquisition time corresponding to each task correction image and each actual positioning position of the optional accessories corresponding to each task correction image, obtain the number of times the optional accessories move and the cumulative time when the optional accessories do not move.

[0116] In this embodiment, according to the acquisition time corresponding to each task correction image, any two adjacent frames of current image in each task correction image are determined, and the two actual positioning positions of the optional accessories corresponding to the two frames of current image (that is, the actual positioning positions of the optional accessories relative to the basic accessories) are compared. If they are the same, it means that the optional accessories have not moved. If they are different, it means that the optional accessories have moved.

[0117] In this embodiment, the counter and the timer can be started separately, and whenever the optional accessory is detected to move, the count value of the counter is increased by 1, and the accumulated time of the timer is cleared.

[0118] The count value of the counter is used to indicate the number of times the optional accessory has moved, and the accumulated duration of the timer is used to indicate the accumulated duration during which the optional accessory has not moved.

[0119] Step S1006 : When the number of times the optional accessory is moved exceeds a preset number threshold or the cumulative duration of the optional accessory exceeds a preset duration threshold, operation prompt information of the task operation process is provided.

[0120] In this embodiment, when the count value of the counter exceeds the preset number threshold, or the accumulated time of the timer exceeds the preset time threshold, it can be determined that the user cannot complete the current task operation process, and the task program can give relevant operation prompts to help the user successfully complete the current task operation process.

[0121] To sum up, this embodiment detects the movement status of optional accessories and gives relevant operation prompts in a timely manner when it determines that the user cannot successfully complete the current task operation process, thereby improving the user's task execution experience, increasing the user's interest in task execution, and enhancing the educational effect.

[0122] Figure 11 This is a structural block diagram of a task program execution device according to an exemplary embodiment of the present application. As shown in the figure, the task program execution device 1100 of this embodiment mainly includes:

[0123] The execution unit 1102 is configured to execute the task program and determine the current execution process of the task program.

[0124] The acquisition unit 1104 is connected to the execution device and is used to acquire the distorted image of the current execution process.

[0125] The execution unit 1102 is also used to: perform correction on the distorted image of the current execution process according to the image correction parameters of the task program to obtain the corrected image of the current execution process; perform comparison based on the image detection information of the corrected image and the image reference information of the current execution process, and if the image detection information matches the image reference information, continue to execute the task program until the execution of the task program is completed.

[0126] Optionally, the task program execution device 1100 is, for example, Figure 1 The electronic device 102 (eg, a tablet computer, etc.) is shown. The acquisition unit 1104 is, for example, a camera 1022 integrated on the electronic device 102 .

[0127] Optionally, the task program execution device 1100 further includes: a bracket 106 for positioning the task program execution device; a reflector 108 mounted on the task program execution device 1100 (eg Figure 1 The electronic device 102 shown in FIG. 1 is located adjacent to the acquisition unit 1104 (eg, Figure 1 The camera 1022 shown is used to adjust the viewing range of the acquisition unit 1104.

[0128] Optionally, the task program execution device 1100 further includes: a display unit for displaying the operation prompt of the current execution process. For example, the display unit is Figure 1 Display screen 1024 is shown.

[0129] Optionally, the execution unit 1102 is further configured to determine the image correction parameters of the task program in the following manner:

[0130] Acquire a reference graphic of the task program to obtain a distorted image including a distorted graphic; detect each corner point of the distorted graphic to obtain a plurality of distorted coordinates of the distorted graphic; determine a plurality of reference coordinates and a plurality of coordinates to be calibrated among the plurality of distorted coordinates according to the shooting posture of the distorted image; calibrate the plurality of coordinates to be calibrated according to the reference graphic and the plurality of reference coordinates to obtain a plurality of calibration coordinates of the plurality of coordinates to be calibrated; obtain the image correction parameters of the task program according to a first coordinate set composed of the distorted coordinates and a second coordinate set composed of the reference coordinates and the calibration coordinates.

[0131] Optionally, the execution unit 1102 is further configured to: output an operation prompt of the current execution process according to the current execution process of the task program; wherein the operation prompt of the current execution process matches the image reference information of the current execution process.

[0132] Optionally, the task program includes a task preparation process and at least one task operation process following the task preparation process; the image reference information of the task preparation process includes: a reference viewing range and basic accessory reference information; the basic accessory reference information includes: a reference category of the basic accessory of the task program and a reference positioning position of the basic accessory relative to the actual viewing range of the corrected image; the image reference information of the task operation process includes: optional accessory reference information; the optional accessory reference information includes: a reference category of the optional accessory of the task program and a reference positioning position of the optional accessory relative to the basic accessory.

[0133] Optionally, when the current execution process is the task preparation process, the execution unit 1102 is further used to: detect the corrected image of the task preparation program and determine the actual framing range of the task preparation program; compare the reference framing range and the actual framing range, and if the actual framing range matches the reference framing range, detect the actual category of the basic accessories in the corrected image and the actual positioning position of the basic accessories relative to the actual framing range to obtain basic accessory detection information; if the actual framing range does not match the reference framing range, repeat the step of detecting the corrected image of the task preparation program and determining the actual framing range of the task preparation program; compare the basic accessory detection information and the basic accessory reference information, and if the basic accessory reference information matches the basic accessory reference information, continue the step of executing the task program and determining the current execution process of the task program; if the basic accessory reference information does not match the basic accessory reference information, execute the step of detecting the actual category of the basic accessories in the corrected image and the actual positioning position of the basic accessories relative to the actual framing range to obtain basic accessory detection information.

[0134] Optionally, when the current execution process is any task operation process, the execution unit 1102 is also used to: detect the actual category of the optional accessory in the corrected image and the actual positioning position of the optional accessory relative to the basic accessory to obtain optional accessory detection information; compare the optional accessory reference information and the optional accessory detection information, and if the optional accessory detection information is consistent with the optional accessory reference information, continue the step of executing the task program and determining the current execution process of the task program; if the optional accessory detection information is inconsistent with the optional accessory reference information, execute the step of detecting the actual category of the optional accessory in the corrected image and the actual positioning position of the optional accessory relative to the basic accessory to obtain optional accessory detection information.

[0135] Optionally, the basic accessory includes a base plate having a plurality of blocks of the same size arranged in a matrix; the execution unit 1102 is further configured to detect the actual positioning position of the optional accessory relative to the basic accessory by:

[0136] The block size is determined according to the base plate size of the base plate and the number of rows and columns of the multiple blocks on the base plate; the lateral difference of the optional accessory along the lateral direction and the longitudinal difference along the longitudinal direction are obtained according to the center coordinates of the optional accessory and any vertex coordinate of the base plate; the number of rows and columns of an actual block in which the optional accessory falls among the multiple blocks is obtained according to the lateral difference, the longitudinal difference and the block size.

[0137] Optionally, when the current execution process of the task program is any task operation process, the execution unit 1102 is also used to: obtain multiple task correction images of the task operation process based on a preset time interval; obtain the number of movements of the optional accessory and the cumulative time during which the optional accessory has not moved according to the acquisition time corresponding to each task correction image and the actual positioning position of the optional accessory corresponding to each task correction image; when the number of movements of the optional accessory exceeds a preset number threshold or the cumulative time of the optional accessory exceeds a preset time threshold, provide operation prompt information of the task operation process.

[0138] An embodiment of the present disclosure provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the task program execution method described in each exemplary embodiment of the present disclosure.

[0139] The exemplary embodiments of the present disclosure provide an electronic device including: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores a computer program executable by the at least one processor, wherein the computer program, when executed by the at least one processor, causes the electronic device to perform the task program execution method according to the exemplary embodiments of the present disclosure.

[0140] Please refer to Figure 12 , a block diagram of an electronic device 1200 that can serve as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0141] like Figure 12 As shown, electronic device 1200 includes a computing unit 1201, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1202 or a computer program loaded from a storage unit 1208 into a random access memory (RAM) 1203. Various programs and data required for the operation of device 1200 can also be stored in RAM 1203. Computing unit 1201, ROM 1202, and RAM 1203 are connected to each other via a bus 1204. An input / output (I / O) interface 1205 is also connected to bus 1204.

[0142] Multiple components within electronic device 1200 are connected to I / O interface 1205, including an input unit 1206, an output unit 1207, a storage unit 1208, and a communication unit 1209. Input unit 1206 can be any type of device capable of inputting information into electronic device 1200. Input unit 1206 can receive input numeric or character information and generate key input signals related to user settings and / or function control of the electronic device. Output unit 1207 can be any type of device capable of presenting information and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. Storage unit 1208 may include, but is not limited to, a magnetic disk or an optical disk. Communication unit 1209 allows electronic device 1200 to exchange information / data with other devices via computer networks such as the Internet and / or various telecommunication networks and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver and / or chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0143] The computing unit 1201 can be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1201 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1201 performs the various methods and processes described above. For example, in some embodiments, the task program execution method as described above can be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1208. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 1200 via the ROM 1202 and / or the communication unit 1209. In some embodiments, the computing unit 1201 can be configured to perform the task program execution method described above by any other appropriate means (e.g., by means of firmware).

[0144] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0145] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0146] As used in this disclosure, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., a magnetic disk, an optical disk, a memory, a programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0147] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0148] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0149] A computer system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The client and server relationship arises through computer programs running on the respective computers and having a client-server relationship to each other.

[0150] It should be pointed out that, according to the needs of implementation, the various components / steps described in the embodiments of the present disclosure can be split into more components / steps, or two or more components / steps or partial operations of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present disclosure.

[0151] The above implementation methods are only used to illustrate the embodiments of the present disclosure, and are not intended to limit the embodiments of the present disclosure. Ordinary technicians in the relevant technical field may make various changes and modifications without departing from the spirit and scope of the embodiments of the present disclosure. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of the present disclosure, and the scope of patent protection of the embodiments of the present disclosure should be defined by the claims.

Claims

1. A task program execution method, comprising: executing a task program and determining a current execution progress of the task program; Obtaining a distorted image of the currently executed process, and performing correction on the distorted image of the currently executed process according to image correction parameters of the task program to obtain a corrected image of the currently executed process; performing a comparison based on the image detection information of the corrected image and the image reference information of the currently executed process, and if the image detection information matches the image reference information, continuing the step of executing the task program until the task program is completed; The task program includes a task preparation process and at least one task operation process following the task preparation process; The image reference information of the task preparation process includes: reference viewing range and basic accessories reference information; The basic accessory reference information includes: a reference category of the basic accessory of the task program and a reference positioning position of the basic accessory relative to an actual viewing range of the corrected image; The image reference information of the task operation process includes: optional accessories reference information; The optional accessory reference information includes: a reference category of the optional accessory of the task program and a reference positioning position of the optional accessory relative to the basic accessory; The basic accessory includes a base plate having a plurality of blocks of the same size arranged in a matrix form.

2. The method according to claim 1, wherein The image correction parameters for the task program are determined as follows: Acquiring a reference pattern of the task program to obtain a distorted image including a distorted pattern; detecting corner points of the distorted pattern to obtain a plurality of distorted coordinates of the distorted pattern; Determining, according to the shooting posture of the distorted image, a plurality of reference coordinates and a plurality of coordinates to be calibrated among the plurality of distorted coordinates; calibrating the plurality of coordinates to be calibrated according to the reference pattern and the plurality of reference coordinates to obtain a plurality of calibration coordinates of the plurality of coordinates to be calibrated; Image correction parameters of the task program are obtained according to a first coordinate set consisting of the distorted coordinates and a second coordinate set consisting of the reference coordinates and the calibration coordinates.

3. The method according to claim 1, wherein The method further comprises: Outputting an operation prompt of the current execution process according to the current execution process of the task program; The operation prompt of the currently executed process matches the image reference information of the currently executed process.

4. The method according to claim 1, wherein In a case where the currently executed process is the task preparation process, performing comparison based on the image detection information of the corrected image and the image reference information of the currently executed process, and continuing the step of executing the task program if the image detection information matches the image reference information, includes: detecting a rectified image of the mission preparation procedure to determine the actual viewing range of the mission preparation procedure; Comparing the reference framing range with the actual framing range, if the actual framing range matches the reference framing range, detecting the actual category of the basic accessory in the corrected image and the actual positioning position of the basic accessory relative to the actual framing range to obtain basic accessory detection information; if the actual framing range does not match the reference framing range, repeating the step of detecting the corrected image of the task preparation procedure to determine the actual framing range of the task preparation procedure; Compare the basic accessory detection information and the basic accessory reference information. If the basic accessory reference information matches the basic accessory reference information, continue to execute the task program and determine the current execution process of the task program. If the basic accessory reference information does not match the basic accessory reference information, execute the step of detecting the actual category of the basic accessory in the corrected image and the actual positioning position of the basic accessory relative to the actual viewing range to obtain the basic accessory detection information.

5. The method according to claim 1, wherein In a case where the currently executed process is any task operation process, performing comparison based on the image detection information of the corrected image and the image reference information of the currently executed process, and continuing the step of executing the task program if the image detection information matches the image reference information, includes: detecting an actual category of the optional accessory and an actual position of the optional accessory relative to the basic accessory in the corrected image to obtain optional accessory detection information; Compare the optional accessory reference information and the optional accessory detection information. If the optional accessory detection information matches the optional accessory reference information, continue the step of executing the task program and determine the current execution process of the task program. If the optional accessory detection information does not match the optional accessory reference information, perform the step of detecting the actual category of the optional accessory in the corrected image and the actual positioning position of the optional accessory relative to the basic accessory to obtain the optional accessory detection information.

6. The method according to claim 5, wherein: The actual positioning position of the optional accessory relative to the basic accessory is detected by: determining a block size according to a base plate size of the base plate, a number of rows and a number of columns of the plurality of blocks on the base plate; Obtaining a transverse difference value and a longitudinal difference value of the optional accessory along the transverse direction according to the center coordinate of the optional accessory and any vertex coordinate of the base plate; According to the horizontal difference, the vertical difference, and the block size, the number of rows and the number of columns of an actual block in which the optional accessory falls among the multiple blocks are obtained.

7. The method according to claim 1, wherein In the case where the current execution process of the task program is any task operation process, the method includes: obtaining a plurality of task-corrected images of the task operation process based on a preset time interval; Obtaining, based on the acquisition times corresponding to the corrected images for each task and the actual positioning positions of the optional accessories corresponding to the corrected images for each task, the number of times the optional accessories have moved and the cumulative length of time during which the optional accessories have not moved; When the number of times the optional accessory is moved exceeds a preset number threshold or the cumulative duration of the optional accessory exceeds a preset duration threshold, operation prompt information of the task operation progress is provided.

8. A task program execution device, comprising: an execution unit, configured to execute a task program and determine a current execution process of the task program; an acquisition unit connected to the execution device and configured to acquire the distorted image of the current execution process; The execution unit is further configured to: Correcting the distorted image of the current execution process according to the image correction parameters of the task program to obtain a corrected image of the current execution process; performing a comparison based on the image detection information of the corrected image and the image reference information of the currently executed process, and if the image detection information matches the image reference information, continuing to execute the task program until the execution of the task program is completed; The task program includes a task preparation process and at least one task operation process following the task preparation process; The image reference information of the task preparation process includes: reference viewing range and basic accessories reference information; The basic accessory reference information includes: a reference category of the basic accessory of the task program and a reference positioning position of the basic accessory relative to an actual viewing range of the corrected image; The image reference information of the task operation process includes: optional accessories reference information; The optional accessory reference information includes: a reference category of the optional accessory of the task program and a reference positioning position of the optional accessory relative to the basic accessory; The basic accessory includes a base plate having a plurality of blocks of the same size arranged in a matrix form.

9. The task program execution device according to claim 8, wherein: The task program execution device also includes: a bracket for positioning the task program execution device; A reflector is installed on the task program execution device and adjacent to the acquisition unit, and is used to adjust the viewing range of the acquisition unit.

10. The task program execution device according to claim 8, wherein: The task program execution device also includes: The display unit is used to display the operation prompt of the currently executed process.

11. An electronic device comprising: processor; as well as Memory for storing programs, The program includes instructions, and when the instructions are executed by the processor, the processor executes the task program execution method according to any one of claims 1 to 7.

12. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to enable the computer to execute the task program execution method according to any one of claims 1 to 7.

Images