Information processing device, control method for information processing device, and program

Abstract

To provide more optimal support for program creation using visual programming. [Solution] The interface device 102 assists in program creation by arranging blocks associated with processes. The interface device 102 displays the cost of a process as a cost display 203, corresponding to the block 200. The interface device 102 arranges the block 200 in the block arrangement unit 202 according to instructions from the user. The interface device 102 displays an overall cost display 204 showing the total cost of a series of processes associated with multiple blocks arranged in the block arrangement unit 202, and a cost constraint display 205 showing the constraints that the overall cost must satisfy.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, a control method for an information processing apparatus, and a program.

Background Art

[0002] Generally, software is developed using a dedicated language called a programming language. However, acquisition of a programming language is often difficult, and it may take a huge amount of time for an inexperienced person to develop software. In contrast, in recent years, visual programming that supports software development using a GUI (Graphical User Interface) has attracted attention. When using visual programming, even a beginner can easily and intuitively develop software by connecting blocks representing processes, conditions, etc. Patent Document 1 discloses a technique that enables development of software that satisfies constraint conditions using visual programming. Further, Patent Document 2 discloses a technique related to model construction of a neural network using visual programming. Recently, software equipped with a visual programming function has also appeared for image generation AI (Artificial Intelligence). By using this software, it is possible to generate a desired image by connecting, on a GUI, processes such as a correction instruction and a change in resolution to an image generated by inputting an instruction sentence and a resolution.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] When a program created using visual programming is actually implemented on hardware, it may be required that the program satisfy constraints such as processing time and memory usage. However, the technology described in Patent Document 1 makes it difficult to represent hardware constraints. Furthermore, the technology described in Patent Document 2 makes it difficult to verify whether the constraints are being deviated from before the model is completed.

[0005] In view of the above-mentioned problems, the present invention aims to support program creation using visual programming in a more suitable manner. [Means for solving the problem]

[0006] The information processing device according to the present invention is an information processing device that supports the creation of a program by arranging blocks associated with a process, and is characterized by comprising: a first display means for displaying the cost of the process in association with the block; a block arrangement means for arranging the block in a predetermined display area in response to instructions from a user; and a second display means for displaying the total cost of a series of processes associated with a plurality of blocks arranged in the predetermined display area, and the constraints that the total cost must satisfy. [Effects of the Invention]

[0007] According to the present invention, it becomes possible to support program creation using visual programming in a more suitable manner. [Brief explanation of the drawing]

[0008] [Figure 1] This diagram shows an example of the system configuration. [Figure 2] This diagram provides an overview of the features of visual programming. [Figure 3] This diagram shows an example of data linking processing and cost. [Figure 4]This is a diagram showing an example of cost display. [Figure 5] This diagram provides an overview of the features of visual programming. [Figure 6] This diagram shows an example of a block arrangement section. [Figure 7] This figure shows an example of changing block parameters. [Figure 8] This diagram shows an example of how cost constraints associated with hardware upgrades are displayed. [Figure 9] This diagram illustrates an example of block joining and separating. [Figure 10] This diagram illustrates the automatic sorting and optimization of blocks. [Figure 11] This diagram shows an example of the procedure for optimizing block placement. [Figure 12] This diagram shows an example of the hardware configuration of an information processing device. [Modes for carrying out the invention]

[0009] Preferred embodiments of this disclosure will be described in detail below with reference to the attached drawings. In this specification and the drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant descriptions will be omitted.

[0010] <First Embodiment> Referring to Figure 1, an example of the configuration of a system to which the information processing apparatus according to the first embodiment of this disclosure is applied will be described.

[0011] The storage device 101 stores data linking the processing implemented by the program with the cost of that processing, and information on the placement of blocks (blocks linked to processing) placed in response to user instructions in visual programming. The storage device 101 can be implemented, for example, by a server connected to a network.

[0012] The interface device 102 consists of a display device, an input device, and an arithmetic device. In this embodiment, for the sake of convenience, it is assumed that the interface device 102 is realized as a PC having a touch panel type display in which these components are integrated, but the configuration of the interface device 102 is not necessarily limited thereto. For example, each of the display device, input device, and arithmetic device, which are components of the interface device 102, may be realized as an individual device, such as a combination of a display, a mouse, and a desktop PC.

[0013] The program execution device 103 schematically shows a device that is the main body of executing a target program. In this embodiment, for the sake of convenience, it is assumed that a digital camera is applied as the program execution device 103. The program execution device 103 does not necessarily have to be hardware; for example, a hardware emulator may be applied. Also, in this embodiment, for the sake of convenience, it is assumed that the program created by the user is a program for realizing the subject recognition function implemented in the digital camera.

[0014] The communication functions 104 and 105 schematically show the communication functions that connect the storage device 101, the interface device 102, and the program execution device 103. In this embodiment, for the sake of convenience, it is assumed that the functions of network communication are applied as the communication functions 104 and 105.

[0015] Note that the configuration described above with reference to FIG. 1 is merely an example and does not limit the configuration of the system to which the information processing device according to this embodiment is applied. For example, the system may be realized as a smartphone with a camera function in which the storage device 101, the interface device 102, and the program execution device 103 are integrated. In this embodiment, an information processing device that supports the creation of a program by the user is proposed based on the above configuration.

[0016] Referring to FIG. 12, an example of the hardware configuration of the information processing device (for example, the interface device 102) according to this embodiment will be described. The CPU (Central Processing Unit) 1201 controls the operation of the entire information processing unit by loading and executing the control program stored in the ROM (Read Only Memory) 1203. The RAM (Random Access Memory) 1202 is used as a temporary storage area for the CPU 1201, such as its main memory and work area. The RAM 1202 is also used as an area to load the control program so that it can be executed by the CPU 1201. The input unit 1205 corresponds to an input interface that receives input from the user and is implemented by an input device such as a keyboard or touch panel. The display unit 1206 corresponds to an output interface that presents various information to the user and is implemented by an output device such as a liquid crystal display, and is capable of displaying various data and the results of various processes. Furthermore, the information processing device 1210 can communicate with other devices by connecting to a predetermined network via the communication unit 1204. The memory unit 1207 is a memory area for storing various types of data. The memory unit 1207 can be implemented by a storage device such as an HDD (Hard Disk Drive), flash memory, or various optical media. The processing described later can be achieved, for example, by the CPU 1201 loading programs stored in the ROM 1203 or memory unit 1207 into the RAM 1202 and executing them.

[0017] Referring to Figure 2, an overview of the visual programming function, which supports user program creation, implemented by the information processing device according to this embodiment will be described. Note that only an overview of this function will be provided here; details will be described separately. Figure 2 shows an example of a screen presented to the user via the interface device 102.

[0018] Block 200 is display information associated with the program's processing. The block display unit 201 is a display area in which some or all of the data linked to processing and cost stored in the storage device 101 is displayed as blocks 200. In Figure 2, the portion denoted by reference numeral 203 shows an example of display information (hereinafter also referred to as cost display) that visually identifies the cost of the process associated with the target block. In this embodiment, the execution time of each process is applied as the cost, and this cost is displayed as the vertical length of the block 200 to which the target process is associated.

[0019] The block placement section 202 is a display area for creating programs by the user arranging blocks (for example, block 200) displayed on the block display section 201 using so-called visual programming. Blocks displayed on the block display section 201 are placed on the block placement section 202, for example, by drag-and-drop operations. In Figure 2, the portion denoted by reference numeral 204 shows an example of display information (hereinafter also referred to as the total cost display) that visually identifies the total cost when executing the processes associated with each of the series of blocks arranged in the block arrangement unit 202. In this embodiment, the total length obtained by stacking the multiple blocks arranged in the block arrangement unit 202 vertically corresponds to the total cost. In Figure 2, the area labeled 205 shows an example of display information (hereinafter also referred to as cost constraint display) that visually identifies the cost constraints of the target hardware. To satisfy the constraints of the target hardware, the program is created by arranging blocks so that they fit within the area of ​​the cost constraint display 205 indicated by the dotted line.

[0020] In this embodiment, the blocks to be placed in the block placement section 202 (for example, block 200) are represented in two dimensions, but since no cost constraints are set in the horizontal direction, the blocks and cost constraint display 205 are variable in the horizontal direction. In Figure 2, the part denoted by reference numeral 206 represents a conditional branch. In the processing of the part indicated by conditional branch 206, either process 1 or process 2 is executed. In this embodiment, process 3 following conditional branch 206 is executed after the process with the highest cost (process 1 in the example shown in Figure 2) has finished, out of all the processes that can be executed by conditional branch 206 (process 1 and process 2 in the example shown in Figure 2). Although not shown in Figure 2, the condition of conditional branch 206 may also be displayed, for example, as a callout.

[0021] Figure 2(b) is a flowchart showing a series of processes corresponding to the placement of blocks in the block placement unit 202, as illustrated in Figure 2(a). The flowchart shown in Figure 2(b) may be updated in synchronization with the placement status of blocks in the block placement unit 202, for example, as auxiliary information.

[0022] (Block display section) The block display unit 201 receives some or all of the data linking processes and costs from the storage device 101 and displays it as a block 200. An example of data linking processes and costs will be explained with reference to Figure 3. In this embodiment, it is assumed that the data linking processes and costs is stored in the storage device in RDB (Relational Database) format. Figure 3 shows an example of the data structure of an RDB related to the management of cost-linked data. The RDB exemplified in Figure 3 consists of three columns: process name, process content, and processing time (in ms). In the example shown in Figure 3, data corresponding to each of the five blocks from process 1 to process 5 shown in the block display unit 201 of Figure 2 is registered.

[0023] In this embodiment, as illustrated in Figure 2, all processes managed by the RDB are displayed in the block display unit 201 in the order in which they are registered in the RDB. On the other hand, a filtering function to display blocks for only some processes, or a sorting function to change the display order of blocks may be provided. The block display unit 201 displays rectangular blocks (for example, block 200 shown in Figure 2) based on RDB data. In this embodiment, for example, the processing name is displayed in the center of the rectangular block, and the vertical length of the block is controlled to correspond to the processing time. Although not shown in Figure 2, a control may also be applied to display the processing content corresponding to the block in a pop-up window when the block is selected by a long press operation.

[0024] (Block arrangement section) The block placement unit 202 receives block input from the block display unit 201. In this embodiment, the block placement unit 202 receives block input via drag-and-drop operation. Multiple blocks, each associated with a specific process, are placed in the block placement unit 202. In this embodiment, each of the multiple blocks placed in the block placement unit 202 is targeted sequentially from the top, and the process associated with the targeted block is executed each time.

[0025] Conditional branching may be represented, for example, by arranging multiple blocks horizontally on top of each other, as shown in the conditional branching 206 in Figure 2. Although not shown in Figure 2, the branching conditions may be displayed, for example, as callouts. In addition, as auxiliary information, a flowchart corresponding to the placement status of blocks in the block placement unit 202 may be displayed as shown in Figure 2(b), and the flowchart may be updated synchronously when blocks are placed or deleted in the block placement unit 202. Furthermore, in response to user instructions, information about the blocks placed in the block placement unit 202 may be stored in the storage device 101, and the information stored in the storage device 101 may be read out, and blocks may be placed in the block placement unit 202 based on that information. Instructions from the user regarding the storage of this information and instructions regarding the reading of this information can be received at any time.

[0026] Once the placement of blocks in the block placement unit 202 is complete and a program that satisfies the cost constraint display conditions described later is completed, information about the placed blocks is transmitted to the program execution device 103 via the communication function 105. This makes the program executable on the program execution device 103.

[0027] (Cost display) The cost display 203 shown in Figure 2 indicates the cost associated with executing process 2 linked to block 200. The cost display is linked to the characteristics of the target block (in other words, the characteristics of the process linked to the block). In this embodiment, we focus on the shape of the block, and the vertical length of the block is associated with the processing time, which is the cost of the process linked to the block. Note that the association between the characteristics of the block and the cost of the process linked to the block as exemplified in Figure 2 is merely an example; for example, the horizontal or depth length may be associated with the cost of the target process. To convert from the cost of the target process to the length of the block (for example, the vertical length), the calculation formula shown below as (Equation 1) can be used, for example.

[0028]

number

[0029] In Equation 1, disp, a, and cost represent the display parameter, coefficient, and cost, respectively. The display parameter, for example, the vertical length, is expressed in pixels. Here, using the example of applying processes 1 to 5 shown in Figure 2 and the RDB shown in Figure 3, the specific details of cost display 203 will be explained below. In the example shown in Figure 3, the costs, i.e., processing times, for processes 1 to 5 are 20ms, 12ms, 5ms, 8ms, and 45ms, respectively. Under these assumptions, if the coefficient a=10 setting is applied, the vertical lengths of the blocks corresponding to processes 1 to 5 will be 200 pixels, 120 pixels, 50 pixels, 80 pixels, and 450 pixels, respectively.

[0030] Here, referring to Figure 4, we will explain another example of how to display the cost of each process to the user as a cost display. In the example shown in Figure 4, it is assumed that three processes with processing times of 3ms, 6ms, and 9ms corresponding to the costs are set as the candidates for selection. Cost displays 401 through 403 each show an example of a cost display when the display parameter is an angle. When the coefficient a=10 setting is applied, the angles for cost displays 401 through 403 will be 30 degrees, 60 degrees, and 90 degrees, respectively. Cost indications 411 through 413 each show an example of cost indication when the indication parameter is density. When the coefficient a = 1 / 30 setting is applied, the densities of cost indications 411 through 413 will be 10%, 20%, and 30%, respectively. Using a similar approach, costs may also be expressed by adding or subtracting colors or by transparency. Cost displays 421 to 423 each show an example of cost display when the display parameter is a horizontal texture. When the coefficient a = 1 / 3 setting is applied, the number of textures in cost displays 421 to 423 will be 1, 2, and 3, respectively. Note that, as shown in cost displays 431 to 433, textures in multiple directions may be combined, or the cost may be represented by the periodicity of a dot pattern or checkerboard pattern. Cost displays 441 to 443 each show an example of a cost display where the display parameter is the number of icons (star-shaped icons in the example shown in Figure 4). When the coefficient a = 1 / 3 is set, the number of icons in cost displays 431 to 433 will be 1, 2, and 3, respectively. Note that there are no particular limitations on the shape of the icons, and other shapes other than stars may be used. Also, using a similar approach, costs may be expressed using the color (e.g., color intensity), shape (e.g., number of vertices of a polygon), or facial expression (e.g., difference in mouth angle) of the icons. Furthermore, the process of displaying costs related to the execution of processes linked to blocks, such as the cost display 203 exemplified in Figure 2 and the series of cost displays exemplified in Figure 4, corresponds to an example of the first display process.

[0031] (Total cost display) The overall cost display 204 shown in Figure 2 represents the total cost associated with executing the processes linked to each of the series of blocks placed in the block placement unit 202. The overall cost display is expressed based on the block placement information (in other words, the arrangement of blocks in the block placement unit 202). In this embodiment, since the vertical length of each block is associated with the cost, the placement information indicates the overlap of blocks in the vertical direction, and the overall cost is expressed as the total length of the blocks stacked vertically. In the example shown in Figure 2, the series of blocks stacked vertically are linked to Process 1, Process 2, Process 1, Process 3, and Process 4, respectively. The execution order of these processes is determined according to the order in which the blocks corresponding to each process are stacked from the top of the block placement unit 202. As mentioned above, in branching processes, the cost of the process with the highest cost is applied as the cost of that branching process. Also, as mentioned above, the vertical lengths of the blocks linked to Processes 1 through 5 are 200 pixels, 120 pixels, 50 pixels, 80 pixels, and 450 pixels, respectively. Therefore, the total cost in the example shown in Figure 2 is 200 + 120 + 200 + 50 + 80 = 650 pixels. The above is merely one example; the total length in the horizontal or depth direction when stacking blocks in that direction could also be expressed as the overall cost.

[0032] Here, referring to Figure 4, another example of a method for presenting the user with the overall cost associated with executing the processes linked to each of the series of blocks placed in the block placement unit 202, as an overall cost display, will be described. The overall cost display 404 shows an example where the overall cost is represented by the sum of the angles of overlapping blocks in the angular direction. The angles of cost displays 401 to 403 are 30 degrees, 60 degrees, and 90 degrees, respectively, as mentioned above. Therefore, the angle of the overall cost display 404 is 180 degrees, which is the sum of the angles of cost displays 401 to 403. The overall cost display 414 shows an example of how the overall cost can be represented by a change in color. In the example shown in Figure 4, the cost is represented by the intensity of the color, with the closer to black, the higher the cost. The intensity of cost displays 411 to 413 is 10%, 20%, and 30%, as mentioned above. Therefore, the intensity of the overall cost display 414 is 60%, which is the sum of the intensity of cost displays 411 to 413. Note that the sum of intensity is just one example of a change in color; for example, the overall cost could be represented by adding or subtracting colors, such that blue and red are added together to make purple, or the sum of costs could be converted to any color using a pre-prepared color map. The overall cost display 424 shows an example of how the overall cost can be represented by the blending of textures. In the example shown in Figure 4, the cost is represented by the number of horizontal stripes in the texture, with a higher number of stripes indicating a higher cost. As mentioned above, the number of stripes in cost displays 421 to 423 is 1, 2, and 3, respectively. Therefore, the number of stripes in the overall cost display 424 is the sum of the number of stripes in cost displays 421 to 423, resulting in a texture with 6 stripes. Note that the overall cost can also be represented by combining textures in multiple directions, increasing or decreasing the period of dot patterns or checkerboard patterns, or overlapping textures, as shown in the overall cost display 434. Overall cost display 444 shows an example of how the overall cost is represented by changes in icons. In the example shown in Figure 4, the cost is represented by the number of star-shaped icons, with a higher number of icons indicating a higher cost. As mentioned above, the number of icons in cost displays 441 to 443 is 1, 2, and 3, respectively. Therefore, the number of icons in overall cost display 444 is 6, which is the sum of the number of icons in cost displays 441 to 443. Note that, as with the cost displays, the shape of the icons is not limited to stars; the overall cost may also be represented using the color (e.g., color intensity), shape (e.g., number of vertices in a polygon), or facial expression (e.g., difference in mouth angle) of a face-shaped icon.

[0033] (Cost constraint display) The cost constraint display 205 shown in Figure 2 is display information that visually identifies the constraints that the overall cost must satisfy; in other words, it shows the cost constraints when running a program on the target hardware. In this embodiment, the cost constraint display is displayed on the block arrangement unit 202 based on constraint information acquired from the program execution device 103 via the communication function 105. The method of acquiring the constraint information is not particularly limited; for example, constraint information pre-stored in the storage device 101 may be acquired via the communication function 104. Another example is that the constraint information may be acquired based on instructions received by the interface device 102 from the user.

[0034] In this embodiment, the vertical length of each block is associated with the cost of the process linked to that block, so the cost constraint is represented as a vertical length. In the example shown in Figure 2, as mentioned above, the vertical length representing the total cost (the vertical length of the total cost display 204) is 650 pixels, while the vertical length of the cost constraint display 205 is 1000 pixels. Under these premises, if we focus on the empty portion (the portion where no blocks are placed) of the cost constraint display 205 shown in Figure 2, it becomes possible to place a block with a cost of the remaining 350 pixels. In other words, for processes 1 to 4, there is an affordance that allows a block to be placed in the empty portion of this cost constraint display 205, but for process 5 (450 pixels), there is no affordance that allows a block to be placed in the empty portion of this cost constraint display 205. By displaying it in this way, the user can intuitively understand that it will be difficult to satisfy the constraint by placing the block associated with process 5 in the block placement unit 202. Furthermore, blocks that will be difficult to place in the block placement unit 202 will be highlighted by applying control such as changing the color in the block display unit 201. Please note that the above is merely an example, and the display method for cost constraints may be appropriately changed to match the overall cost display.

[0035] Here, referring to Figure 4, we will explain an example of a method for presenting to the user the constraints that the overall cost must satisfy as a cost constraint display. Cost constraint display 405 is a cost constraint display corresponding to the overall cost display 404. In the example shown in Figure 4, cost constraint display 405 indicates that the cost constraint is 300 degrees. Cost constraint display 415 is a cost constraint display corresponding to the overall cost display 414. In the example shown in Figure 4, cost constraint display 415 indicates that the cost constraint is a color density of 80%. In this case, the cost constraint display may be displayed superimposed on the overall cost display, as in cost constraint display 405, or it may be displayed alongside the overall cost display, as shown as cost constraint display 415. Note that the cost constraint displays exemplified hereafter will be displayed superimposed on the overall cost display. Cost constraint display 425 is a cost constraint display corresponding to the overall cost display 424. In the example shown in Figure 4, cost constraint display 425 indicates that the cost constraint is a texture with 9 stripes. Also, similar to the overall cost display, cost constraints may be expressed by combining textures in multiple directions, as shown in cost constraint display 435, by increasing or decreasing the period of a dot pattern or checkerboard pattern, or by overlapping textures and changing the color density. Cost constraint display 445 is a cost constraint display corresponding to the overall cost display 444. In the example shown in Figure 4, cost constraint display 445 indicates that the cost constraint is represented by nine star-shaped icons. As with the overall cost display, the shape of the icons is not limited to stars; cost constraints may also be represented using the color (e.g., color intensity), shape (e.g., number of vertices in a polygon), or facial expression (e.g., differences in mouth angle) of the icons. The processing related to the display of the overall cost display 204 and cost constraint display 205 exemplified in Figure 2, and the series of processing related to the overall cost display and cost constraint display exemplified in Figure 4, are examples of the second display processing.

[0036] (Effects of the first embodiment) As mentioned above, in this embodiment, the cost of a process implemented by the program is linked to the characteristics of the block representing that process. Taking the cost display based on the vertical length of the block exemplified in this embodiment as an example, the total cost is represented by the total vertical length of the stacked series of blocks, and the cost constraint display is also based on the vertical length. By applying this configuration, for example, when trying to add a new process to an existing process, it becomes possible to estimate the total cost quickly and easily. As a specific example, if the current total cost is represented by 650 pixels and the cost constraint is 1000 pixels, the user is presented with a space of 350 pixels where blocks linked to the process can be placed. This allows the user to estimate, for example, that they could add up to two processes if each process costs around 150 pixels. As described above, according to this embodiment, users can generate programs to cause the target device to perform desired processing through visual programming, while more intuitively understanding the cost constraints.

[0037] <Second Embodiment> A second embodiment of this disclosure describes an example of configuration and processing when there are two or more types of costs. Figure 5 is a diagram showing an overview of the visual programming function that supports the creation of programs by a user, which is realized by the information processing device according to this embodiment, and shows an example of a screen when there are two types of costs. In the example shown in Figure 5, the execution time of the process and the memory usage rate are the two types of costs. The block 500, block display unit 501, block arrangement unit 502, and cost constraint display 505 shown in Figure 5 correspond to the block 200, block display unit 201, block arrangement unit 202, and cost constraint display 205 in the example shown in Figure 2.

[0038] Furthermore, cost displays 503 and 504 show examples of display information that visually identifies the cost of the process associated with the target block. In this embodiment, the execution time and memory usage rate of each process are applied as costs, and each of these costs is displayed as the vertical and horizontal lengths of the block to which the target process is associated. Specifically, the vertical length of the block (e.g., cost display 503) is associated with the execution time of the process, and the horizontal length (e.g., cost display 504) is associated with the memory usage rate. Furthermore, in the example shown in Figure 5, the relationships between prerequisite processes are indicated by arrows, as indicated by the reference numeral 506. Also, in the example shown in Figure 5, the start of a branching process is represented by a branching arrow, as indicated by the reference numeral 507, and the end of a branching process is represented by a merging arrow, as indicated by the reference numeral 508. Furthermore, reference numeral 509 indicates a cost constraint modification area, which will be described in detail later.

[0039] The above is merely an example; for instance, as illustrated in Figure 6, costs may also be represented using lengths in the vertical, horizontal, and depth directions. The block 600, block display unit 601, block placement unit 602, and cost constraint display 605 shown in Figure 6 correspond to the block 500, block display unit 501, block placement unit 502, and cost constraint display 505 in the example shown in Figure 5. Furthermore, a wider range of costs may be represented by combining elements such as angle, color changes, texture blending, and icon changes.

[0040] (Display of total cost based on memory usage) In the block arrangement section 502 shown in Figure 5, the display of the total cost relative to the processing execution time (vertical length) is substantially the same as the example described in the first embodiment with reference to Figure 2. On the other hand, the total cost relative to the memory usage rate (horizontal length) changes depending on the time.

[0041] Here, we will explain the overall cost of memory usage using the period from time t1 to t3 shown in Figure 5 as an example. The memory usage for processes 1 to 4 is assumed to be 60%, 35%, 50%, and 25%, respectively. The memory usage cost constraint is assumed to be 100%. Focusing on time t1, process 1a is being executed, and memory usage is 60%. Focusing on time t2, processes 2a and 1b are executed in parallel, and the total memory usage of these processes is 95%. Focusing on time t3, a branch process is executed in parallel with process 3b. In this branch process, either process 2b or process 4a is executed. If process 2b is executed in the branch process, the total memory usage at time t3 will be 85%. On the other hand, if process 4a is executed in the branch process, the total memory usage at time t3 will be 75%. To create a program that can run on hardware, it is required that blocks be arranged in such a way that the cost constraints of memory usage are met at all times.

[0042] (Area for modifying cost constraints) Figure 5 illustrates the branching process from the start 507 to the end 508 of the branch. In branching, some of the multiple processes are executed. In the example shown in Figure 5, it is conceivable that process 2b is executed, or that process 4b is executed after process 4a. Focusing on time t3, the processes that can be executed at that time are process 3b, process 2b, and process 4a. The sum of the memory usage of these processes is 110%, making it difficult to satisfy the constraint. However, at time t3, either the pair of process 3b and process 2b, or the pair of process 3b and process 4a, is actually executed, so the maximum sum of memory usage is 85%. To represent this, a cost constraint modification area is provided. Reference numeral 509 indicates an example of a cost constraint modification area.

[0043] Focusing on the branching process from the start of branching 507 to the end of branching 508, the maximum memory usage of the branching process before the end of process 2b is 35%, and the maximum memory usage of the branching process after the end of process 2b is 25%. The cost constraint modification area 509 is modified so that the area shown as a cost constraint is expanded by an area equivalent to 25% before the end of process 2b and 35% after the end of process 2b, taking into account the maximum memory usage during branching.

[0044] The above-described method of displaying branching processes is merely an example; the processes during branching may be displayed overlapping in the depth direction, or the block arrangement section 502 may be labeled with "Branching Process 1," etc., and the content of Branching Process 1 may be displayed separately. If the content of the branching process is displayed separately, the constraint modification area does not need to be used.

[0045] (Effects of the second embodiment) As described above, according to this embodiment, when creating a program that takes into account multiple costs, it is possible to achieve substantially the same effects as in the first embodiment described above for each of those multiple costs. Furthermore, as explained with reference to Figure 5, it is possible to create a program that involves parallel execution processing using visual programming in a more suitable manner.

[0046] <Variation example: Parameter change section and processing result display section> Each block corresponding to a process may have modifiable parameters. Parameter modification of a block will be explained with reference to Figure 7. In this modified example, the information processing device accepts the selection of a block whose parameters are to be modified via the block placement unit. Once a block is selected, the information processing device displays the parameter modification unit, which will be described later.

[0047] For example, Figure 7 shows an example of a parameter change section when detecting the pupils of a dog. The slider 701 is configured to move laterally, as indicated separately by the numeral 711. When moved to the right, the number of detection frames increases, and when moved to the left, the number of detection frames decreases. Block 702 shows an example of a selected block, and the cost (e.g., processing time and memory usage) increases or decreases according to the increase or decrease in the number of detection frames.

[0048] The processing result display unit 703 shows an example of a display area where the results of the processing associated with block 702 are displayed. In the example shown in Figure 7, the processing result display unit 703 displays a still image of a dog that has been prepared in advance by the user. The processing result display unit 703 may also display multiple still images or moving images, or it may display visualized audio data or text data. The processing result display 704 shows an example of display information indicating the execution result of the process associated with block 702. In the example shown in Figure 7, based on the detection result of the dog's pupils, a frame indicating the detected pupil area is displayed as the processing result display 704. Note that the processing result display 704 shown in Figure 7 is merely an example, and the display method may be appropriately changed depending on the target process. As a specific example, when the results of segmentation processing are displayed, the detected area may be filled in. Also, when the results of image generation processing are displayed, the generated image itself may be displayed.

[0049] Reference numerals 712, 713, and 714 indicate examples of the block, processing result display unit, and processing result display when the slider is moved to the state shown by reference numeral 711, respectively. Depending on the movement of the slider, the processing result display changes from the state indicated by reference numeral 704 to the state indicated by reference numeral 714. In the example shown in Figure 7, the cost of the block increases in conjunction with the increase in the number of detected frames, and the number of detected frames also increases. In addition, the cost display of placed blocks may be updated in real time in conjunction with the block placement unit (for example, the block placement unit 202 shown in Figure 2). Furthermore, although this embodiment describes an example in which one process is selected, multiple processes may be selected and multiple parameters may be changeable.

[0050] This modified version allows users to modify the parameters of each process, enabling fine-tuning of the program. Furthermore, users can monitor the impact on cost constraints and processing results in real time while modifying parameters, allowing for efficient program fine-tuning. Furthermore, the display of the processing result 704 on the processing result display unit 703 as illustrated in Figure 7, which is an example of the third display process, corresponds to the display of the results of the processing associated with a block.

[0051] <Variations: Hardware configuration and hardware parameters> Figure 8(a) shows a program that does not satisfy the cost constraint, where the block associated with process 4 extends beyond the cost constraint display 801. If a program created in this way does not satisfy the cost constraint, it is possible to satisfy the cost constraint by changing the program's processes or process parameters, but it is also possible to relax the cost constraint by changing the hardware configuration.

[0052] For example, a situation may arise where the program execution device 103 shown in Figure 1 is changed to a more high-performance one. If the program execution device 103 is changed, the interface device 102 obtains constraint information from the program execution device 103 via the communication function 105 and updates the cost constraint display based on that constraint information.

[0053] The cost constraint display 802 shown in Figure 8(b) is an example of a cost constraint display updated in accordance with a hardware upgrade. In the example shown in Figure 8(b), the horizontal size of the cost constraint display 802 has increased compared to the previous cost constraint display 801 due to the change to hardware with a larger memory capacity. As a result, it has become possible to execute processes 1 and 2 in parallel, and all blocks can be contained within the cost constraint display 802.

[0054] Changes to the program execution device are not limited to changes to the entire device, but may also include changes to the hardware configuration through the replacement or addition of parts. Furthermore, cost constraints may be simulated even if the program execution device is not actually changed, but is virtually modified. Hardware parameters controlling the program execution device may also be changed. Examples of hardware parameter changes include changes to the upper limits of power consumption and heat generation.

[0055] As described above, this modified version allows users to intuitively understand the changes in constraints that accompany hardware changes. As a concrete example, if the process a user is trying to implement does not meet the constraints, they can use the aforementioned simulator to consider changing the hardware. This also applies when the hardware parameters of the program execution device are changed.

[0056] <Example: Combining and separating blocks> Refer to Figure 9 to explain the joining and separating of blocks. In Figure 9(a), reference numeral 901 indicates an example of the block placement unit before the block joining operation. The user selects multiple blocks from the block placement unit 901 and performs the joining operation on those multiple blocks. In Figure 9(b), reference numeral 902 indicates an example of the block placement unit after the joining operation. In the example shown in Figure 9, process 1 and process 2 are joined to form process A, and the two processes 3 and process 4 are joined to form process B. As shown in Figure 9(b) with reference numeral 903, linked blocks are displayed in the block display section. In the example shown in Figure 9(b) with reference numeral 903, the block before linking has been deleted, but it may also be left in place.

[0057] Furthermore, it is possible to select separable blocks from among the blocks arranged in the block arrangement section 902 shown in Figure 9(b) and separate them into multiple blocks. Once the separation operation has been performed on all the blocks arranged in the block arrangement section 902 shown in Figure 9(b), the system will return to the state before the joining operation, as shown in Figure 9(a).

[0058] The above describes an example of how blocks may be joined or separated based on user selection; however, the joining or separation process may also be performed according to pre-defined rules. That is, the interface device 102 may join two or more blocks into one block, or separate a block formed by joining two or more blocks, when certain conditions (e.g., instructions from the user or pre-defined rules) are met. Furthermore, while this modified example shows an example where the joined block is a rectinia polygon formed by connecting multiple rectangles, it may also be approximated as a rectangle encompassing all the blocks to be joined.

[0059] As described above, this modification allows users to link together, for example, multiple recurring processing patterns, thereby enabling them to create programs efficiently. Furthermore, by preparing linking rules in advance for beginners, intermediate users, and advanced users, it becomes possible to provide a program creation environment that suits users with diverse skill levels.

[0060] <Modified example: Automatic block organization and optimization> Refer to Figure 10 to explain automatic block arrangement and optimization. Figure 10(a) is a flowchart showing an example of a process to be programmed. In the example shown in Figure 10(a), processes 1a to 3a and processes 1b to 3b are processed in parallel. Figures 10(b), 10(c), and 10(d) each show an example of a block placement section where the vertical axis represents processing time and the horizontal axis represents memory usage as a cost display. Specifically, Figure 10(b) shows an example of a block placement section when a user places blocks. Figure 10(c) shows an example of a block placement section where the block arrangement is automatically arranged. Figure 10(d) shows an example of a block placement section where the block arrangement is optimized.

[0061] First, let's explain the automatic rearrangement of block placement. Here, it is assumed that blocks are rearranged according to predetermined rules. Specifically, in this modified example, the predetermined rules are the two procedures described below as Procedure 1 and Procedure 2. Procedure 1 is the procedure to place all blocks as far to the left as possible, stopping when they touch the left edge of the cost constraint display or the right edge of horizontally overlapping blocks. Procedure 2 is the procedure to place all blocks as far up as possible, stopping when they touch the top edge of the cost constraint display or the bottom edge of vertically overlapping blocks, similar to Procedure 1.

[0062] Figure 10(c) shows an example of what happens when the aforementioned rearrangement is applied in the state shown in Figure 10(b). Let's focus on process 3b in Figure 10(c). Process 3b has space to be placed upwards. However, as shown in the flowchart in Figure 10(a), process 3b is a subsequent process to process 2b, and is therefore placed below the bottom edge of process 2b.

[0063] The above-mentioned automatic block sorting is merely an example, and the method of automatic sorting may be arbitrarily determined by the user. Furthermore, a placement lock function may be provided that allows the user to set whether or not to include each block in the automatic sorting process. Controls may also be applied to visualize the automatic sorting process or to display the differences before and after the automatic sorting.

[0064] Next, we will explain block placement optimization. Here, as an example, we will assume that block placement optimization is performed by applying the bottom-left method, which is a common solution method for two-dimensional packing problems. Referring to Figure 11, an example of the optimization procedure will be described in detail. First, the interface device 102 classifies all blocks according to the number of prerequisite processes. The blocks shown in Figure 11 labeled 1101, 1102, and 1103 represent processes with no prerequisite processes, processes with one prerequisite process, and processes with two prerequisite processes, respectively. In the example shown in Figure 11, the naming convention for processes is to write the prerequisite process at the beginning and assign the last character alphabetically starting from 'a'. Processes a through f are processes that have no prerequisite processes. Process ca is a process that has prerequisite process c. Process da is a process that has prerequisite process d. There are also two processes that have prerequisite process e, which are shown as process ea and process eb. Process caa and process daa are processes that have prerequisite processes ca and da, respectively.

[0065] The symbol 1104 in Figure 11 shows an example of a cost constraint representation. A block without preconditions is placed inside the cost constraint representation 1104 using the bottom-left method. Note that while the bottom-left method generally prioritizes the bottom left, in Figure 11, the top left is prioritized. Once all blocks without prerequisite processing have been placed, the area from the top of the cost constraint display 1104 to the bottom of the placed series of blocks (the position indicated by reference numeral 1105 in Figure 11) is designated as a non-placement area.

[0066] Next, one prerequisite processing block is placed inside the cost constraint display 1104, excluding the non-placement area, using the bottom-left method. Once all blocks of one prerequisite processing block have been placed, the area from the top of the cost constraint display 1104 in Figure 11 to the bottom of the placed series of blocks (the position indicated by reference numeral 1106 in Figure 11) is designated as the non-placement area. Next, the two prerequisite processing blocks are placed inside the cost constraint representation 1104, excluding the areas where placement is not possible, using the bottom-left method. In the example shown in Figure 11, there are only two prerequisite processing blocks, so the optimization ends here. However, if there are blocks with more prerequisite processing, the optimization process is repeated until all blocks are placed.

[0067] Figure 10(d) shows an optimized arrangement of the blocks shown in Figure 10(c). In the example shown in Figure 10(d), processes 2a and 2b are parallelized, resulting in more processing time compared to Figure 10(c). Further optimizations could be made by automatically adding processes to the parts with available processing time, or by automatically adjusting the parameters of blocks in the parts with available memory usage. Thus, the interface device 102 according to this modified example optimizes the arrangement of the multiple blocks arranged in the block arrangement section according to the characteristics of the processes associated with each of the multiple blocks (for example, the number of prerequisite processes).

[0068] As described above, this modification allows users to focus on implementing the more important logic parts of the program without having to perform detailed layout optimization. Furthermore, by using the layout lock function and visualizing the layout optimization process, it becomes possible to prevent or correct unintended layout optimizations. By using the layout optimization function described above, it becomes possible to create a lean program, and by adding processing or adjusting parameters in areas where cost margins have been freed up, it becomes possible to explore even more advanced processing.

[0069] <Other Embodiments> The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0070] Furthermore, the disclosure of this embodiment includes the following configurations, methods, and programs. (Configuration 1) An information processing device that supports the creation of a program by arranging blocks associated with a process, comprising: a first display means for displaying the cost of the process in association with the block; a block arrangement means for arranging the block in a predetermined display area in response to instructions from a user; and a second display means for displaying the total cost of a series of processes associated with a plurality of blocks arranged in the predetermined display area, and the constraints that the total cost must satisfy. (Configuration 2) The information processing apparatus according to Configuration 1, wherein the first display means displays the cost of the process associated with the block in relation to the characteristics of the block. (Configuration 3) The information processing apparatus according to Configuration 2, characterized in that the characteristics of the block include at least one of the shape, color, transparency, texture, and icon of the block. (Configuration 4) The information processing apparatus according to any one of Configurations 1 to 3, characterized in that the total cost of a series of processes associated with a plurality of blocks arranged in the predetermined display area is expressed based on the arrangement information of the plurality of blocks. (Configuration 5) The information processing device according to Configuration 4, wherein the arrangement information is information indicating the overlapping of multiple blocks in at least one of the vertical, horizontal, depth, and angular directions, and the total cost of a series of processes associated with the multiple blocks arranged in the predetermined display area is displayed in such a way that it includes at least one of the sum of the total length of the stacked multiple blocks, the sum of the angles of the stacked multiple blocks, the color change corresponding to the overlapping blocks, and the blending of textures. (Configuration 6) The information processing apparatus according to any one of Configurations 1 to 5, wherein the process has a changeable parameter, and the first display means changes the display of the cost of the block to which the process is associated when the parameter of the process is changed, in accordance with the change in the parameter. (Configuration 7) A third display means for displaying the processing results of a series of processes associated with a plurality of blocks arranged in the predetermined display area, wherein the third display means changes the processing results of the series of processes in accordance with the change in the parameter when the parameter is changed. An information processing device according to configuration 6, characterized in that... (Configuration 8) The information processing apparatus according to any one of Configurations 1 to 7, wherein the second display means controls the display of constraints that the overall cost must satisfy, according to the hardware configuration of the device that performs the series of processes. (Configuration 9) The information processing apparatus according to any one of Configurations 1 to 8, wherein the second display means controls the display of constraints that the overall cost must satisfy according to the hardware parameters of the device that causes the series of processes to be executed. (Configuration 10) The information processing apparatus according to any one of Configurations 1 to 9, wherein the second display means applies at least one of the following when a predetermined condition is met: a process of combining two or more blocks displayed in the predetermined display area, and a process of separating one block formed by combining two or more blocks. (Configuration 11) The information processing device according to any one of Configurations 1 to 10, wherein the block arrangement means rearranges a plurality of blocks arranged in the predetermined display area according to predetermined rules. (Configuration 12) The information processing apparatus according to any one of Configurations 1 to 11, wherein the block arrangement means optimizes the arrangement of blocks according to the characteristics of the processing associated with each of the plurality of blocks arranged in the predetermined display area. (Method 1) A control method for an information processing device that supports the creation of a program by arranging blocks associated with a process, comprising: a first display step of displaying the cost of the process in association with the block; a block arrangement step of arranging the block in a predetermined display area in response to instructions from a user; and a second display step of displaying the total cost of a series of processes associated with a plurality of blocks arranged in the predetermined display area, and the constraints that the total cost must satisfy. (Program 1) A program for causing a computer to function as an information processing device that assists in the creation of a program by arranging blocks associated with processing, the information processing device comprising: a first display means for displaying the cost of processing in association with the blocks; a block arrangement means for arranging the blocks in a predetermined display area in response to instructions from a user; and a second display means for displaying the total cost of a series of processing associated with a plurality of blocks arranged in the predetermined display area, and the constraints that the total cost must satisfy. [Explanation of Symbols]

[0071] 102 Interface device 202 Block arrangement section 203 Cost Display 204 Total cost display 205 Cost Constraint Display

Claims

1. An information processing device that assists in program creation by arranging blocks associated with processing, A first display means that displays the cost of the process in correspondence with the aforementioned block, A block placement means that arranges the block in a predetermined display area in response to instructions from the user, A second display means for displaying the total cost of a series of processes associated with a plurality of blocks arranged in the predetermined display area, and the constraints that the total cost must satisfy. An information processing device characterized by having

2. The information processing apparatus according to claim 1, characterized in that the first display means displays the cost of the process associated with the block in relation to the characteristics of the block.

3. The information processing apparatus according to claim 2, characterized in that the characteristics of the block include at least one of the shape, color, transparency, texture, and icon of the block.

4. The information processing apparatus according to claim 1, characterized in that the total cost of a series of processes associated with a plurality of blocks arranged in the predetermined display area is expressed based on the arrangement information of the plurality of blocks.

5. The aforementioned arrangement information is information indicating the overlapping of multiple blocks in at least one of the following directions: vertical, horizontal, depth, and angular. The total cost of the series of processes associated with the multiple blocks arranged in the predetermined display area is displayed to include at least one of the following: the sum of the total length of the stacked blocks, the sum of the angles of the stacked blocks, the color changes corresponding to the overlapping blocks, and the blending of textures. The information processing apparatus according to claim 4, characterized in that

6. The above process has modifiable parameters, The first display means, when the parameters of the process are changed, changes the display of the cost of the block to which the process is associated in accordance with the change in the parameters. The information processing apparatus according to claim 1, characterized in that

7. It has a third display means for displaying the processing results of a series of processes associated with a plurality of blocks arranged in the predetermined display area, The third display means modifies the processing result of the series of processes in accordance with the change in the parameter when the parameter is changed. The information processing apparatus according to claim 6, characterized in that

8. The information processing apparatus according to claim 1, characterized in that the second display means controls the display of constraints that the overall cost must satisfy, according to the hardware configuration of the device that performs the series of processes.

9. The information processing apparatus according to claim 1, characterized in that the second display means controls the display of constraints that the overall cost must satisfy according to the hardware parameters of the device that causes the series of processes to be executed.

10. The information processing apparatus according to claim 1, wherein the second display means is characterized by applying at least one of the following when a predetermined condition is met: a process of combining two or more blocks displayed in the predetermined display area, and a process of separating one block formed by combining two or more blocks.

11. The information processing apparatus according to claim 1, wherein the block arrangement means rearranges a plurality of blocks arranged in the predetermined display area according to predetermined rules.

12. The information processing apparatus according to claim 1, wherein the block arrangement means optimizes the arrangement of blocks according to the characteristics of the processing associated with each of the plurality of blocks arranged in the predetermined display area.

13. A control method for an information processing device that assists in program creation by arranging blocks associated with processing, A first display step that displays the cost of the process in association with the aforementioned block, A block placement step in which the block is placed in a predetermined display area in response to instructions from the user, A second display step that displays the total cost of a series of processes associated with a plurality of blocks arranged in the predetermined display area, and the constraints that the total cost must satisfy. A control method for an information processing device, characterized by including the following:

14. Computers, An information processing device that assists in program creation by arranging blocks associated with processing, A first display means that displays the cost of the process in correspondence with the aforementioned block, A block placement means that arranges the block in a predetermined display area in response to instructions from the user, A second display means for displaying the total cost of a series of processes associated with a plurality of blocks arranged in the predetermined display area, and the constraints that the total cost must satisfy. A program for causing an information processing device to function, characterized by having the following features.

Images