Processing device and processing system

Abstract

To properly execute the process selected by the operation. [Solution] A processing device according to an embodiment includes a display unit, an operation unit, and a processing unit. The display unit displays blocks representing devices that perform input or output, as well as blocks representing processes involving input and output, and further displays selected blocks. The operation unit enables operations to select blocks on the display unit, connect a connector representing output in one of the selected blocks with a connector representing input in another block, and group multiple selected blocks. The processing unit assigns processes represented by the blocks to processing resources according to the connection status between the selected blocks and the grouping, then determines the order in which the processing resources will be executed, and executes the processes assigned to the processing resources according to the order.

Description

[Technical Field]

[0001] SUMMARY OF THE INVENTION The embodiments disclosed herein and in the drawings relate to processing devices and processing systems. [Background technology]

[0002] There is a GUI (Graphical User Interface) technology that allows a user to select boxes representing input / output devices, image processing, etc. on the screen of a PC or the like, and then connect the selected boxes with lines to execute a process that, for example, processes an image taken by a camera and displays the results on a display. Normally, specialized programming is required to perform such processing, but by using the above-mentioned GUI technology, users can achieve the processing they envision without any programming whatsoever.

[0003] In detail, the threads to which processes are actually assigned and the order in which those threads are executed are determined based on the boxes representing devices and processes and the state of the connections between those boxes, and each thread is executed in that order. A thread is a processing unit of the CPU in a PC (Personal Computer). However, optimizing the processes to be executed poses several challenges.

[0004] For example, there is the problem of the long time it takes from the start of processing until the results are available. For example, when processing streaming video captured from a camera frame by frame, if the processing is slow, the next frame will be sent while a certain frame is being received and processed. As a result, real-time processing cannot be achieved, and the frames to be processed will pile up, eventually resulting in overflow.

[0005] Such processing time is greatly affected by whether the user-selected processing is executed sequentially in a single thread or divided into multiple threads and executed in parallel. Furthermore, when distributing processing across multiple machines, there is a need not only to speed up the overall processing, but also to "perform the same processing at a lower cost." In other words, when distributing processing across multiple sub-PCs connected to a main PC via a network, it is necessary to minimize the total computational costs of the sub-PCs while also taking into account the reduction of processing time. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] “[Japanese Subtitles] #11 Connecting Batch Nodes Flame 2021”, [online], July 21, 2021, YouTube NGC Corporation, [Retrieved July 2, 2024], Internet <URL:https: / / www.google.com / search?q=autodesk+flame+batch&rlz=1C1DKGK_jaJP1007JP1008&oq=&gs_lcrp=EgZjaHJvbWUqCQgBECMYJxjqAjIJCAAQIxgnGOoCMgkIARAjGCcY6gIyCQgCECMYJxjqAjIJCAMQIxgnGOoCMgkIBBAjGCcY6gIyCQgFECMYJxjqAjIJCAYQIxgnGOoCMgkIBxAjGCcY6gLSAQkzMTQ3ajBqMTWoAgiwAgE&sourceid=chrome&ie=UTF-8#fpstate=ive&vld=cid:fae20b36,vid:Rk4cunQTByI,st:0> Summary of the Invention [Problem to be solved by the invention]

[0007] One of the problems that the embodiments disclosed in this specification and the drawings aim to solve is to properly execute a process selected by an operation. [Means for solving the problem]

[0008] A processing device according to an embodiment includes a display unit, an operation unit, and a processing unit. The display unit displays blocks representing devices that perform input or output, and blocks representing processes involving input and output, and further displays selected blocks. The operation unit enables operations to select blocks on the display unit, connect a connector representing output in one of the selected blocks with a connector representing input in another block, and group multiple selected blocks. The processing unit allocates processes represented by the blocks to processing resources according to the connection status between the selected blocks and the grouping, then determines an order in which the processing resources will be executed, and executes the processes allocated to the processing resources according to the order. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a schematic diagram showing an outline of processing performed by an image processing system according to an embodiment. [Figure 2] FIG. 1 is a block diagram showing the configuration of an image processing system according to an embodiment. [Figure 3] FIG. 4 is a view showing a display example of a first GUI screen according to the first embodiment. [Figure 4] 4 is a flowchart showing a procedure for allocating processes to threads and executing the processes according to the first embodiment. [Figure 5] FIG. 10 is a view showing a display example of a second GUI screen according to the first embodiment. [Figure 6] FIG. 10 is a view showing a display example of a third GUI screen according to the second embodiment. [Figure 7] 11 is a flowchart showing a procedure for allocating threads to PCs and executing them according to the third embodiment. [Figure 8] 13 is a time chart showing a comparative example of distributed allocation of threads according to the third embodiment. [Figure 9]10 is a flowchart showing a procedure for optimizing a combination of a process and a PC using supervised learning according to the fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of a processing apparatus and a processing system will be described in detail with reference to the drawings.

[0011] [Processing overview] First, an overview of the processing of the image processing system 1 according to the embodiment will be described. Details will be described from FIG. 2 onwards. FIG. 1 is a schematic diagram showing an overview of the processing of the image processing system 1 according to the embodiment. In the image processing system 1, a user describes an intended processing on a PC screen using a GUI. As shown in FIG. 1, the user selects boxes representing devices or batch processing (hereinafter simply referred to as "processing") and connects the boxes with lines with arrows indicating input / output relationships between the devices or processes. A box is an example of a block. The image processing system 1 executes the processing indicated by the box while inputting / outputting data to / from the device indicated by the box, depending on the box selected by the user and the connection state between the boxes. This allows the user to freely realize various processing without programming.

[0012] Input devices connectable to the image processing system 1 include, for example, a webcam, a USB (Universal Serial Bus) connected camera, a server, a sensor, a microphone, etc. A webcam and a USB connected camera are examples of cameras capable of capturing images of a subject. A sensor detects physical quantities. Output devices connectable to the image processing system 1 include, for example, a display, an alarm device, a speaker, an LED (Light-Emitting Diode) light, etc. Alarm devices include warning lights and loudspeakers. These multiple devices can be handled in an integrated manner. In other words, a user can combine measurement devices such as cameras and sensors and display devices such as displays and LED lights as one process (input / output process) in the same way as combining processes. Although external devices 10 such as input devices and output devices (see FIG. 2) are outside the configuration of the image processing system 1, these external devices 10 may be included as components of the image processing system 1.

[0013] Meanwhile, the types of processing executed by each box include image processing, person tracking (including face recognition), numerical conditions, image display, text-to-speech conversion, etc. Machine learning can also be used to execute processes such as image processing, person tracking (including face recognition), and text-to-speech conversion. For example, the image processing system 1 executes processing based on image data of a subject acquired from a webcam or physical quantities acquired from a sensor, and displays the results of the processing on a display. The image processing system 1 is an example of a processing unit. The display is an example of a display unit.

[0014] Furthermore, even if different languages ​​and platforms, such as C++, Python, and GPU (Graphics Processing Unit), are mixed in the image processing system 1, the processes running on each can be efficiently combined in the background and processed in an integrated manner. In particular, data is transferred between different languages ​​and platforms, such as between C++ and Python, or between C++ and GPU processing, so different languages ​​and platforms can be handled simultaneously. This allows the user to achieve the desired processing without being aware of differences between languages ​​or platforms.

[0015] When actually executing a process, the image processing system 1 assigns the process indicated by the box to threads to be executed sequentially and threads to be executed in parallel based on the box selected on the GUI screen and the connection state between the boxes. Furthermore, in the image processing system 1, multiple threads are executed by one PC, or multiple threads are assigned to multiple PCs and executed (distributed processing).

[0016] FIG. 2 is a block diagram showing the configuration of an image processing system 1 according to an embodiment. As shown in FIG. 2, in the image processing system 1, a main PC 2 and sub PCs 3 and 4 are configured to be able to communicate with each other via a network N. The number of sub PCs is not limited to two, and may be three or more. The main PC 2 is a PC operated by a user, and may execute processes collectively, or may assign processes to the sub PCs 3 and 4 or other sub PCs for execution. The sub PCs 3 and 4 and other sub PCs execute the processes assigned by the main PC 2. The main PC 2 is an example of a main processing device. The sub PCs 3 and 4 and other sub PCs are examples of sub-processors.

[0017] The main PC 2 includes a camera 20, a display 21, an input interface 22, a network interface 23, a processing circuit 24, a memory 25, and a device interface 26.

[0018] The camera 20 photographs a subject, creates an image of the subject, and stores the image data in the memory 25. The display 21 displays various types of information. For example, the display 21 outputs data generated by the processing circuit 24, a GUI for accepting various operations from the user, and the like. Any of a variety of displays can be used as the display 21, as appropriate. For example, the display 21 can be a liquid crystal display (LCD), a cathode ray tube (CRT) display, an organic electroluminescence display (OLED), or a plasma display. The display 21 is an example of a display unit.

[0019] The input interface 22 accepts various input operations from the user, converts the accepted input operations into electrical signals, and outputs the electrical signals to the processing circuit 24. For example, the input interface 22 accepts operations from the user, such as selecting a box on the GUI screen displayed on the display 21, connecting boxes, and inputting parameters into boxes. As the input interface 22, for example, a mouse, keyboard, trackball, switch, button, joystick, touchpad, touch panel display, etc. can be used as appropriate. The input interface 22 is an example of an operation unit.

[0020] The network interface 23 connects the main PC 4 to the sub-PCs 3 and 4 via the network N so that they can communicate with each other.

[0021] The processing circuitry 24 controls the overall operation of the main PC 2 in response to electrical signals of input operations output from the input interface 43. The processing circuitry 24 executes predetermined processing based on the input data and outputs the results of the processing. For example, the processing circuitry 24 has, as hardware resources, a processor such as a CPU, MPU, or GPU, and memory such as ROM or RAM. The processing circuitry 24 executes predetermined processing using a processor that executes a program loaded in the memory. The processing circuitry 24 is an example of a processing unit.

[0022] The memory 25 is a storage device such as an HDD (Hard Disk Drive), an SSD (Solid State Drive), or an integrated circuit storage device that stores various information. In addition to an HDD or SSD, the memory 25 may be a portable storage medium such as a CD (Compact Disc), a DVD (Digital Versatile Disc), a BD (Blu-ray (registered trademark) Disc), or a flash memory. The memory 25 may also be a drive device that reads and writes various information from and to semiconductor memory elements such as flash memory and RAM.

[0023] The device interface 26 inputs and outputs data to and from the external device 10. The external device 10 is a web camera, a sensor, a display, an alarm device, etc. The web camera transfers image data at, for example, more than 1000 frames per second to the main PC 2 in real time as streaming data.

[0024] The sub-PC 3 includes a display 31, an input interface 32, a network interface 33, a processing circuit 34, and a memory 35. Each component is the same as the component of the main PC 2. The configuration of the sub-PC 4 is the same as the configuration of the sub-PC 3, so a detailed description will be omitted.

[0025] [First embodiment] In the first embodiment, an example of processing by one main PC 2 will be described. The main PC 2 is an example of a processing device. For example, when the main PC 2 receives video input from a webcam of the external device 10, it extracts human faces from the video and displays the extracted results on the display 21. In addition, when the number of faces is greater than 0, the main PC 2 causes an alarm device of the external device 10 to issue an alarm that a person who should not be present in that location is present.

[0026] FIG. 3 is a diagram showing a display example of a first GUI screen SC1 according to the first embodiment. The GUI screen SC1 is displayed on the display 21 or the display of the external device 10. Initially, a toolbox TB is displayed at the bottom of the GUI screen SC1. The toolbox TB contains many boxes representing devices that input or output data, as well as many boxes representing processes involving data input and output. While the boxes are arranged in 3 rows and 7 columns in FIG. 3, this arrangement is not limiting.

[0027] The user selects the boxes "Webcam," "People Tracking," "Image Display," "Numeric Value a>b," and "Text to Speech." Specifically, the user uses the mouse of the input interface 22 to drag and drop each box from the toolbox TB to an empty area above the toolbox TB on the GUI screen SC1. As a result, each selected box is further displayed in the empty area.

[0028] For example, after a user connects a webcam to the main PC 2, the webcam starts capturing images by dragging and dropping the "Webcam" box on the screen of the display 21. The "Person Tracking," "Image Display," "Numerical Value a>b," and "Text to Speech Conversion" boxes correspond to person tracking processing, image display processing, numerical value a>b determination processing, and text to speech conversion processing by the processing circuit 24.

[0029] As shown in Figure 3, connectors are attached to the left and right of each box. The connector attached to the left of a box indicates the input to the device or process represented by the box. The connector attached to the right of a box indicates the output from the device or process represented by the box. In other words, the device or process represented by a box performs processing on the input represented by the connector attached to the left of the box, and produces the output represented by the connector attached to the right of the box.

[0030] On the GUI screen SC1, the user uses the mouse of the input interface 22 to connect a connector indicating output in one of the selected boxes to a connector indicating input in another box. For example, the user connects the following four combinations with lines: the output of "webcam" and the input of "people tracking," the preview of "people tracking" and the input of "image display," the number of faces in "people tracking" and the a in "numerical value a>b," and the output of "numerical value a>b" and the trigger for "text-to-speech conversion."

[0031] A series of processes described on the GUI screen SC1 will be explained in accordance with the above-described connection state between the boxes. The web camera of the external device 10 outputs a captured two-dimensional image to the person tracking process.

[0032] The person tracking process inputs a character string indicating parameters related to face tracking in advance. The person tracking process inputs a 2D image from a webcam, applies rectangular frames to people's faces through image processing using the above parameters, and counts the number of such rectangular frames (i.e., the number of faces). The person tracking process then outputs the original 2D image, a preview including the rectangular frames, and the number of faces.

[0033] In addition, the person tracking process may involve inputting a string of characters indicating parameters for tracking parts other than the face (e.g., eyes, mouth, etc.) in advance, adding a rectangular frame to the part in question in the two-dimensional image input from the webcam, and outputting the two-dimensional image, a preview including the rectangular frame, and the number of parts.

[0034] In the image display process, a preview of a two-dimensional image is input from the person tracking process, and the preview is displayed on the display 21 or the display of the external device 10.

[0035] The process for determining whether the value a>b is a process where the threshold value 0 for the determination is input in advance as the value b. Next, the process for determining whether the value a>b is input as the number a from the person tracking process. Then, the process for determining whether the value a>b is output as the truth value of the determination formula a>b to the text-to-speech conversion process. Since the value b is set to 0, if the number a, which is the number of faces, is 1 or greater, the truth value is true. On the other hand, if the value a is 0 or less, the truth value is false.

[0036] The toolbox TB may include a box that indicates a numeric output process that changes and outputs the numeric value b according to the passage of time (for example, the time of day). After dragging and dropping the box, the user may connect the output of the numeric value b of the box with the input of the numeric value b of "Numeric value a>b" with a line. For example, since one security guard patrols a specific location during a certain time period, the numeric output process may output the numeric value b=1 as the threshold value 1 for determining whether to issue an alarm. The "Numeric value a>b" process inputs a number from the numeric output process and updates the numeric value b. Since the numeric value b is set to 1, if the numeric value a, which is the number of faces, is 2 or greater, the truth value is true. On the other hand, if the numeric value a is 1 or less, the truth value is false.

[0037] In the text-to-speech conversion process, a character string indicating a warning message is input in advance. The warning message is, for example, "There is someone there. Please be careful." The text-to-speech conversion process acquires a truth value as a trigger from a determination process of whether the numerical value a>b. If the truth value is true, the text-to-speech conversion process converts the character string input in advance into speech and outputs the speech to the loudspeaker speaker of the external device 10. The text-to-speech conversion process may also cause a warning light to be outputted to a warning light. If the truth value is false, the text-to-speech conversion process does not perform any processing. Note that "the truth value is true" is an example of a predetermined condition. The loudspeaker speaker and the warning light are examples of warning devices. When a predetermined condition is satisfied, the processing circuitry 24 may display a warning message on the multi-screen display, or may send a control signal to a robot arm to operate the robot arm.

[0038] For example, on a display connected to a PC with a built-in graphics board, a user connects a box representing image output processing to a box representing image processing by the graphics board. This enables image processing to be performed using an application (e.g., CUDA) compatible with the graphics board (e.g., an NVIDIA board). CUDA (Compute Unified Device Architecture) is a general-purpose parallel computing platform (parallel computing architecture) and programming model for GPUs.

[0039] Fig. 4 is a flowchart showing the procedure for allocating processes to threads and executing them according to the first embodiment. Fig. 4 shows the procedure in which the main PC 2 extracts processes from, for example, a GUI screen SC1 as shown in Fig. 3, allocates the processes to multiple threads, determines the order of the threads, and executes the processes. Below, the processing of the main PC 2 will be described according to Fig. 4, with reference to Fig. 3 as an example of a GUI screen.

[0040] In step S1, the processing circuitry 24 recognizes boxes and the wiring between the boxes from the GUI screen. In the example of Fig. 3, the processing circuitry 24 extracts the boxes "Webcamera," "Person Tracking," "Image Display," "Numeric Value a>b," and "Text-to-Speech Conversion." Then, based on the wiring, the processing circuitry 24 recognizes that the order of processing the boxes is as follows: "Webcamera" is followed by "Person Tracking," "Image Display" and "Numeric Value a>b" are followed by "Numeric Value a>b," and "Text-to-Speech Conversion."

[0041] In step S2, the processing circuit 24 determines whether the boxes are connected by only one line. If the boxes are connected by only one line (YES in step S2), the processing circuit 24 proceeds to step S3. Note that the number of boxes connected by only one line is not limited to two, and may be three or more. If the boxes are not connected by only one line (NO in step S2), the processing circuit 24 skips step S3 and proceeds to step S4. In the example of FIG. 3, the boxes for "Webcam" and "Person Tracking" and the boxes for "Numerical Value a>b" and "Text-to-Speech Conversion" are connected by only one line. Therefore, the processing circuit 24 proceeds to step S3.

[0042] In step S3, the processing circuitry 24 assigns to one thread the processes corresponding to the boxes connected by only one line, which are executed sequentially. In the example of FIG. 3, the processing circuitry 24 assigns the webcam-related process and the person tracking process to thread A, and the process of determining whether a number a>b is true and the text-to-speech conversion process to thread B. The processes assigned to threads A and B are executed sequentially in each thread. As described above, when a connector indicating an output in one box and a connector indicating an input in another box are connected by a single line, the processing circuitry 24 executes the process indicated by the one box and then executes the process indicated by the other box.

[0043] In step S4, the processing circuit 24 determines whether the output of one box is connected to the inputs of multiple boxes. If the output of one box is connected to the inputs of multiple boxes (YES in step S4), the processing circuit 24 proceeds to step S5. If the output of one box is not connected to the inputs of multiple boxes (NO in step S4), the processing circuit 24 skips steps S5, S6, and S7 and proceeds to step S8. In the example of FIG. 3, the "person tracking" box is connected to the "image display" box and the "numeric value a>b" box. Therefore, the processing circuit 24 proceeds to step S5.

[0044] In step S5, processing circuitry 24 determines whether shared resources used among multiple boxes overlap. Shared resources are resources shared among multiple boxes, such as input devices, data in memory 25, output devices, etc. When shared resources are used among multiple boxes, they are used exclusively under the control of processing circuitry 24. If the shared resources used by the multiple boxes overlap (YES in step S5), the processing circuitry 24 proceeds to step S6. If there are no shared resources used by the multiple boxes or if there are no overlapping resources (NO in step S5), the processing circuitry 24 proceeds to step S7. In the example of FIG. 3, the process of determining whether or not the numerical value a>b and the text-to-speech conversion process have already been assigned to thread B, so the processing circuitry 24 determines whether or not there are shared resources used between the image display process and the process of thread B. The display used in the image display process and the speaker used in the text-to-speech conversion process of thread B are different, that is, the shared resources used between the two processes do not overlap, so the processing circuitry 24 proceeds to step S7.

[0045] In step S6, the processing circuitry 24 assigns to one thread the processes corresponding to the multiple boxes connected to the output of one box that are executed sequentially. That is, when the connectors indicating multiple outputs in one box are connected to the connectors indicating inputs in multiple boxes, the processing circuitry 24 executes each process sequentially if the processes indicated by the multiple blocks use the same resources.

[0046] In step S7, the processing circuitry 24 assigns processes corresponding to multiple boxes connected to the output of one box to separate threads for parallel execution. In the example of Fig. 3, the processing circuitry 24 assigns image display processing to thread C, which is different from thread B, for parallel execution with thread B. As described above, when the connectors representing multiple outputs in one box are wired to the connectors representing inputs in multiple boxes, and when the processes represented by the multiple boxes use different resources, the processing circuitry 24 executes the processes represented by the multiple blocks in parallel.

[0047] In step S8, the processing circuitry 24 determines the order of the threads and executes the processes assigned to each thread according to the order. In the example of Fig. 3, the processing circuitry 24 first executes the webcam process and person tracking process of thread A, and then executes the numerical value a>b process and text-to-speech conversion process of thread B and the image display process of thread C in parallel.

[0048] Fig. 5 is a diagram showing a display example of the second GUI screen SC2 according to the first embodiment. Fig. 5 shows an example in which processing corresponding to three boxes cannot be executed in parallel using three threads, but is successfully processed when the processing is executed sequentially using one thread by grouping the boxes.

[0049] As shown in Fig. 5, GUI screen SC2 displays boxes for "Webcamera," "Edge detection," and "Image synthesis" selected by the user. The boxes for "Webcamera," "Edge detection," and "Image synthesis" correspond to processes related to the webcam, edge detection processing by the processing circuit 24, and image synthesis processing. The output of a two-dimensional image from "Webcamera" is input to "Edge detection" and "Image synthesis." Furthermore, the output of a two-dimensional image from "Edge detection" is input to "Image synthesis."

[0050] A series of processes described on GUI screen SC2 will be explained in accordance with the wiring state between the boxes as described above. The webcam of the external device 10 outputs a captured 2D image to edge detection processing and image synthesis processing. The edge detection processing inputs the 2D image captured by the webcam, detects edges from the 2D image, and generates an image showing the edges. The edge detection processing then outputs the generated image showing the edges to image synthesis processing. The image synthesis processing inputs the original 2D image captured by the webcam, inputs the image showing the edges from the edge detection processing, and synthesizes the two input images. Here, in the image synthesis processing, the 2D image input from the webcam and the 2D image that was the source of the edge image must be the same.

[0051] However, if the processing circuit 24 assigns three processes to three threads and executes them in parallel, after executing the webcam-related process, the edge detection process and the image synthesis process will be executed in parallel. In this case, when the edge detection process outputs an edge image to the image synthesis process, the image synthesis process will input a 2D image from the webcam that is earlier than the 2D image in which the edge was detected. As a result, in the image synthesis process, the 2D image input from the webcam and the 2D image that was the source of the edge image will be misaligned.

[0052] Therefore, the user performs an operation to group multiple boxes, "Webcamera," "Edge Detection," and "Image Synthesis," on the GUI screen SC2 using the mouse of the input interface 22. The grouping operation may be, for example, an operation to select all boxes to be grouped by left-clicking the mouse, or an operation to surround all of the boxes with a rectangle. In response to such an operation, the processing circuit 24 assigns even three processes to one thread and executes them sequentially. According to the above, when multiple boxes among the selected boxes are grouped, the processing circuit 24 assigns the processes indicated by the multiple boxes to one thread, and then sequentially executes the processes indicated by the multiple boxes assigned to the thread.

[0053] That is, the webcam processing, edge detection processing, and image synthesis processing are executed in sequence. As a result, the image synthesis processing starts after the edge detection processing is completed, so in the image synthesis processing, the 2D image input from the webcam matches the original 2D image from which the edges were detected. Therefore, image misalignment in the image synthesis processing can be eliminated. Although the above example shows how processes represented by boxes are assigned to threads, the embodiment is not limited to this example. For example, processes represented by boxes may be assigned to processing resources. A processing resource is a physical entity that executes a process, and may include a thread, which is a logical unit to which a process is assigned in each entity. An entity is a processing entity such as a CPU, a GPU, or a cloud.

[0054] [Effects of the first embodiment] According to the above, the processing circuit 24 assigns the processes indicated by the boxes to threads according to the wiring state between the selected boxes and the grouping, and then determines the order in which the threads will be executed, and executes the processes assigned to the threads according to that order.

[0055] This allows the user to select multiple boxes from the toolbox on the GUI screen, connect them with a line, or group them together to assign processes corresponding to multiple boxes to the same thread and execute them sequentially.Also, by connecting the output of one box to the input of multiple boxes, processes corresponding to multiple boxes can be assigned to separate threads and executed in parallel.

[0056] Therefore, the GUI allows users to easily create the functions they envision, and the thread allocation and processing order are optimized to ensure the fastest possible internal processing. For example, it is now possible to process streaming video captured by a webcam at high speeds of 33 msec or less per frame.

[0057] Second Embodiment In the second embodiment, an explanation will be given of the operation and processing for performing machine learning on a single main PC 2. In machine learning, there are multiple parameters that the user must adjust. On the main PC 2 of the image processing system 1, the user can easily perform an operation for executing processing that adopts the optimal combination while changing various combinations of multiple parameters.

[0058] FIG. 6 is a diagram showing a display example of a third GUI screen SC3 according to the second embodiment. The GUI screen SC3 is displayed on the display 21 of the main PC 2 or the display of the external device 10. The user selects each of the boxes: "CSV (Comma Separated Value) Import," "Test Learning Split," "Linear Regression," two "Model Prediction," "R2 Score," and two "Metric Plot." Note that the processing of each box included in the GUI screen SC3 in FIG. 6 is executed by a Python program.

[0059] Next, the user connects the following 12 combinations with lines on GUI screen SC3. That is, the user connects the data frame output by "CSV Import" with the data frame input by "Test Learning Split," the data frame input by "Model Prediction 2," and the data frame of examples input by "Metric Plot 2." A data frame is a data structure that corresponds to two-dimensional data. For example, if the data frame is in the form of a matrix with m rows and n columns, there will be m combinations of n pieces of data.

[0060] The user connects the training data frame output by "Test Training Split" with the data frame input by "Linear Regression." Then, the user connects the testing data frame output by "Test Training Split" with the data frame input by "Model Prediction 1," the data frame of examples input by "Metric Plot 1," and the data frame of examples input by "R2 Score."

[0061] The user connects the machine learning model output by "Linear Regression" with the machine learning models input by "Model Prediction 1" and "Model Prediction 2." The user connects the data frame output by "Model Prediction 1" with the data frame of prediction results input by "R2 Score" and the data frame of prediction results input by "Metric Plot 1." The user connects the data frame output by "Model Prediction 2" with the data frame of prediction results input by "Metric Plot 2."

[0062] A series of processes described on the GUI screen SC3 will be explained in accordance with the above-mentioned connection state between boxes.

[0063] The CSV reading process inputs text data with a file name, a character indicating the separator that separates elements in the text data (usually a comma =”,”), and a boolean value indicating the presence or absence of an index, which is the line number in the text data. Next, the CSV reading process creates a data frame from the input text data. The detailed procedure is explained below.

[0064] First, the text data is searched from the top left character to the right, and when a comma is found, the string up to the character before that is extracted as an element. Next, the search is started from the character following the comma, and when a comma is found again, the string up to the character before that is extracted as an element. This process is repeated until a line break is found. The final element is the string from the character following the last detected comma to the character just before the line break. The string consisting of the multiple elements extracted at this point is made the first line of the data frame. Then, the next line is searched, and the multiple strings extracted become the second line of the data frame. This process is repeated for all lines contained in the text data.

[0065] Then, the CSV reading process outputs the created data frame to the test learning division process, model prediction process 2, and metric plot process 2.

[0066] The test learning partitioning process inputs a test size, which is a double-precision floating-point number, in advance. The test size is the size of the data frame for testing. The test learning partitioning process inputs a data frame from the CSV reading process and divides the input data frame into a data frame for testing and a data frame for learning according to the test size. In detail, a data frame equal to the test size is extracted for testing, and the remaining data frame is used for learning. The test learning partitioning process outputs the data frame for testing to model prediction process 1, metric plot process 1, and R2 score process. The test learning partitioning process outputs the data frame for learning to linear regression process.

[0067] In linear regression processing, strings indicating X, Y, and other arguments are input in advance. X indicates an explanatory variable, which is a variable that represents the cause. Y indicates a target variable, which is a variable that represents the result. Other arguments are written as *args, which indicates remaining arguments without keyword specification, allowing for flexible response when the number of arguments is variable. In linear regression processing, the training data frame is input from the test training split processing, and linear regression analysis is performed using X, Y, and other arguments to create a machine learning model. The machine learning model referred to here is a regression model, and is, for example, expressed as Equation 1 below. Y = W0X0+W1X1+W2X2+ +W m X m ··· Formula 1

[0068] Then, the linear regression process outputs the created machine learning model to model prediction processes 1 and 2.

[0069] The model prediction process 1 performs predictions using a regression model on a test-size data frame. That is, the model prediction process 1 inputs the explanatory variable X in advance. The model prediction process 1 inputs the machine learning model from the linear regression process in advance. The model prediction process 1 inputs the test data frame from the test learning split process, and creates a data frame of prediction results using the explanatory variable X and the machine learning model. The model prediction process 1 outputs the created data frame to the R2 score process and the metric plot process 1.

[0070] The model prediction process 2 performs predictions using a regression model on the data frame output by the CSV reading process. That is, the model prediction process 2 inputs the explanatory variable X in advance. The model prediction process 2 inputs the machine learning model from the linear regression process in advance. The model prediction process 2 inputs the test data frame from the CSV reading process, and creates a data frame of prediction results using the explanatory variable X and the machine learning model. The model prediction process 2 outputs the created data frame to the metric plot process 2.

[0071] The R2 score process inputs the example data frame from the test learning split process and the prediction result data frame from the model prediction process 1. The R2 score process compares the objective variables contained in the example data frame with the objective variables contained in the prediction result data frame to evaluate the prediction accuracy and outputs the evaluation result as a double-precision floating-point number.

[0072] Metric plot process 1 inputs a character string indicating other arguments in advance. Metric plot process 1 inputs a data frame of examples from test learning division process, and inputs a data frame of prediction results from model prediction process 1. Metric plot process 1 uses the other arguments to create a plot comparing the examples and prediction results.

[0073] Metric plot process 2 inputs a character string indicating other arguments in advance. Metric plot process 2 inputs a data frame of examples from the CSV reading process, and inputs a data frame of prediction results from model prediction process 2. Metric plot process 2 uses the other arguments to create a plot comparing the examples and prediction results.

[0074] Next, we will explain a specific example of machine learning in the main PC2. For example, when measuring how the head of a dummy doll moves during a collision test, black and yellow markers attached to the doll's face are tracked (point tracking). The procedure for the first approach is shown below.

[0075] (1) Set the size of the marker in advance. (2) In the first frame, a full screen search is performed to find the marker. (3) In the second frame, search 100 pixels around the marker found in the first frame.

[0076] As mentioned above, the program is designed to narrow the search range as much as possible, centering on the marker found in the first frame, to enable fast detection.Then, by using GUI operations, it is possible to achieve the same processing without programming.

[0077] Additionally, newer methods can be used. Below is a procedure for the second approach, which uses deep learning processing to track the head of a dummy.

[0078] (1) Set the size of the marker in advance. (2) Detect people or faces, where deep learning processing is used. (3) As a result of the person or face detection process, the coordinate values ​​of a rectangle surrounding the detected person or face are output. (4) A mask indicating the target of processing is generated from the coordinate values ​​of the rectangle. (5) The marker is searched for within the target area indicated by the mask.

[0079] The output results include the number of faces found and the coordinate values ​​of each face (the coordinates of the rectangle surrounding the face's position). In addition to the webcam image, detailed tracking is possible by inputting facial feature information and the parts of the face you want to search for.

[0080] [Effects of the second embodiment] This allows for high-speed marker detection by narrowing down the target area.Furthermore, in addition to being used for tracking, it can also be used in combination with AI processing and conventional image processing, such as detecting a human face in a street interview and blurring only the surrounding area (to protect privacy).

[0081] Third Embodiment In the third embodiment, we will explain optimization when multiple PCs execute thread processing. "Optimization" here means distributing multiple processes to multiple PCs for efficient execution. The main PC 2 optimally distributes the processes to speed up the entire process while ensuring the processing performance required for high-quality image processing, and to reduce the usage fees for the sub-PCs 3 and 4.

[0082] Fig. 7 is a flowchart showing the procedure for assigning threads to each PC and executing them according to the third embodiment. As shown in Fig. 2, the PCs that execute thread processing in the image processing system 1 are the main PC 2 and the sub PCs 3 and 4. It is assumed that the threads and their execution order have been determined according to the flowchart in Fig. 4.

[0083] The following description will be given assuming this configuration and threads, but the system configuration according to the third embodiment is not limited to this. For example, there may be three or more sub-PCs. Furthermore, when processing is distributed to multiple PCs, even processing that uses common resources can be executed in parallel. Therefore, processing assigned to sequentially executed threads in the flowchart of FIG. 4 may be assigned to separate threads that can be executed in parallel.

[0084] In step S11, the processing circuit 24 of the main PC 2 calculates the total amount of calculation for the processes assigned to each thread. Since the processes assigned to one thread are executed sequentially, the amount of calculation for the thread is the total amount of calculation for each process. Then, the processing circuit 24 stores the amount of calculation for each thread in the memory 25.

[0085] In step S12, the processing circuit 24 calculates the amount of input / output data for each thread. The input and output data for each thread may be ignored when the main PC 2 processes the data. However, when the sub-PCs 3 and 4 process the data, the data is transferred between the main PC 2 and the sub-PCs 3 and 4 via the network N, and the calculated data is used to take into account the transfer time. The processing circuit 24 then stores the amount of input / output data for each thread in the memory 25.

[0086] In step S13, the processing circuit 24 acquires the processing performance of the main PC2 and stores the processing performance in the memory 25. The processing time of the thread in the main PC2 is calculated from the calculation amount of the thread and the processing performance of the main PC2.

[0087] In step S14, the processing circuit 24 obtains the processing performance, network communication bandwidth, and usage fee per unit time of the sub-PCs 3 and 4, and stores these numerical values ​​in the memory 25. The processing time for the thread in the sub-PCs 3 and 4 is calculated from the calculation amount of the thread and the processing performance of the sub-PCs 3 and 4. The data transfer time for the thread between the main PC 2 and the sub-PCs 3 and 4 is calculated from the amount of input / output data for the thread and the network communication bandwidth. The cost for the actual use of the sub-PCs 3 and 4 is calculated from the processing time and data transfer time for the thread in the sub-PCs 3 and 4 and the usage fees for the sub-PCs 3 and 4.

[0088] In step S15, the processing circuit 24 calculates the transfer time and processing time for each combination of each thread and the main PC 2 and the sub-PCs 3 and 4. In the example of Fig. 3, when threads A, B, and C are assigned to the main PC 2 and the sub-PCs 3 and 4, the processing time of the main PC 2 and the transfer time and processing time of the sub-PCs 3 and 4 are calculated.

[0089] In step S16, the processing circuit 24 extracts combinations in which the transfer time and processing time of each PC are equal to or less than a threshold value. Here, the transfer time and processing time are the time from the start to the end of the entire process, that is, the time from when the main PC 2 acquires the first data to when it acquires the final data. For example, if sub-PCs 3 and 4 execute different threads in parallel, the time from the start of the earlier one to the end of the later one will be the overall transfer time and processing time.

[0090] In step S17, processing circuitry 24 calculates the costs of sub-PCs 3 and 4 for the combinations extracted in step S16, and then identifies the combination that minimizes the costs.

[0091] In step S18, the processing circuit 24 assigns each thread to the main PC 2 and the sub-PCs 3 and 4 in accordance with the combinations identified in step S17, and causes the processing of each thread to be executed.

[0092] According to the above, the processing circuitry 24 assigns processes indicated by the boxes to threads according to the wiring status between the selected boxes and the grouping, then determines the order in which the threads will be executed and instructs the main PC 2 and sub PCs 3 and 4 to execute the processes assigned to the threads according to the order. The processing circuitry 24 acquires information including the processing performance of the main PC 2 and sub PCs 3 and 4, and further instructs the main PC 2 and sub PCs 3 and 4 to execute the processes based on the information. The above information includes data transmission and reception performance between the main PC 2 and sub PCs 3 and 4. The above information includes the usage fees for sub PCs 3 and 4.

[0093] The processes in steps S15 to S17 are merely an example of a procedure for minimizing the transfer time, processing time, and costs of sub-PCs 3 and 4, and other processing procedures may also be used. For example, the processing circuitry 24 may calculate the costs of sub-PCs 3 and 4 for all combinations without performing the process in step S16. Alternatively, the processing circuitry 24 may first calculate the costs of sub-PCs 3 and 4 for all combinations, extract combinations for which the costs are equal to or less than a threshold, and identify from among the combinations the combination that minimizes the transfer time and processing time.

[0094] FIG. 8 is a time chart showing a comparative example of distributed allocation of threads according to the third embodiment. FIG. 8(a) shows a time chart when three threads that can be executed in parallel are assigned to the main PC 2 and the sub PCs 3, 4, and 5. As shown in FIG. 8(a), the main PC 2 executes the processing of the three threads sequentially. Meanwhile, the sub PCs 3, 4, and 5 receive data from the main PC 2, execute the processing of the threads, and then transmit the data. In the example of FIG. 8(a), even though the sub PCs 3, 4, and 5 execute the processing of the three threads in parallel, the processing time of the main PC 2 is shorter because of the data transmission and reception between the main PC 2 and the sub PCs 3, 4, and 5. Therefore, in this case, three threads are assigned to the main PC 2. Note that when the processing time of one thread is long, the effect of parallel execution comes into play, and the processing time of the sub PCs 3, 4, and 5 is shorter.

[0095] FIG. 8(b) shows a time chart for a case in which three threads that can be executed in parallel are assigned to sub-PC 100, whose computational cost is 1 million yen, and sub-PCs 20A, 20B, and 20C, whose computational cost is 200,000 yen. Assume that sub-PC 100, costing 1 million yen, has processing performance approximately five times that of sub-PC 20, costing 200,000 yen. As shown in FIG. 8(b), sub-PC 100 sequentially receives, processes, and transmits data for each of the three threads. Meanwhile, sub-PCs 20A, 20B, and 20C receive data from main PC 2, then execute thread processing in parallel, and then transmit the data to main PC 2. In the example of FIG. 8(b), both threads require approximately the same processing time. The computational cost in this case is 200,000 yen x 3 = 600,000 yen, which is cheaper than the 1 million yen cost of sub-PC 100, so three threads are assigned to sub-PCs 20A, 20B, and 20C. If the processing time of one thread becomes shorter, the effect of parallel execution will be diminished, and the processing time by the sub-PC 100 will become shorter. In this case, there is a trade-off between prioritizing processing time and calculation cost.

[0096] [Fourth embodiment] In the fourth embodiment, an example of improving the efficiency of distributed processing through machine learning will be described. Fig. 9 is a flowchart showing a procedure for optimizing the combination of processing and PC using supervised learning according to the fourth embodiment. Below, a process for suppressing the processing time, utilization rate, and temperature of a PC will be described. Utilization rate is the ratio of the time that the CPU in a PC is operating to a certain unit time.

[0097] In step S21, the processing circuit 24 records the type of processing, the amount of data to be processed, the PC that performed the processing, the processing time, usage rate, and temperature of the PC at that time. That is, the processing circuit 24 acquires this data and stores it in memory 25. The types of processing include person tracking, text-to-speech conversion, audio data output, and streaming data image display. The processing time of the sub-PCs 3 and 4 includes the time required for data transfer between them and the main PC 2.

[0098] In step S22, the processing circuit 24 trains a supervised learning model using the type of processing, the amount of data to be processed, and the PC that executed the processing stored in step S21 as explanatory variables, and the processing time, utilization rate, and temperature of the PC at that time as objective variables. This makes it possible to obtain predicted values ​​for the processing time, utilization rate, and temperature of the PC at that time from the type of processing, the amount of data to be processed, and the PC that executes the processing. The supervised learning model is an example of a machine learning model.

[0099] In step S23, the processing circuit 24 acquires information about a plurality of processes and a plurality of PCs.

[0100] In step S24, the processing circuit 24 identifies all combinations of multiple processes and multiple PCs to identify the optimal combination of processes and PCs. Note that the process of acquiring and displaying image data is exclusive to the main PC 2 to which the webcam and display are connected, and therefore is not included in the search.

[0101] In step S25, the processing circuitry 24 generalizes the supervised learning model to predict the total processing time and the utilization rate and temperature of each PC for each combination identified in step S24, based on the type of processing, the amount of data to be processed, and the PC executing that processing. If the processing of each PC is executed sequentially, the processing circuitry 24 adds up the processing times of each PC and adds the sum to the total processing time. If the processing of each PC is executed in parallel, the processing circuitry 24 adds up the time from the start of the earliest processing to the end of the latest processing to the total processing time.

[0102] In step S26, the processing circuit 24 identifies the optimal combination of processes and PCs based on the overall processing time, utilization rate and temperature of each PC predicted in step S25, specifically, so as to minimize the overall processing time, utilization rate and temperature of each PC.

[0103] In order to improve the accuracy of the supervised learning model, the network communication bandwidth between the main PC 2 and the sub-PCs 3 and 4 may be added as an input parameter.

[0104] [Effects of the fourth embodiment] According to the above, when there are multiple processes and multiple PCs connected to each other via a network, it is possible to estimate the processing time, usage rate, and temperature of the executing PC according to the combination of PCs executing each process. Therefore, it is possible to determine to which PCs multiple processes should be distributed in order to speed up the processes and save power (load balance).

[0105] For example, if you have three PCs and one of them is heavily used and has a high temperature (which indicates that a high-load process is being performed), then this PC will not be allowed to perform any processing, and the remaining two PCs will each perform half of the processing.

[0106] By incorporating such a judgment function, it becomes possible to schedule distributed processing that executes workflow-based processing quickly and with low load, by referring to information on currently available computing resources.

[0107] According to at least one of the embodiments described above, the process selected by the operation can be executed appropriately.

[0108] Although several embodiments have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, modifications, and combinations of embodiments can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, as well as within the scope of the invention and its equivalents as defined in the claims. [Explanation of symbols]

[0109] 1...Image processing system 2. Main PC 10. External Devices 20...Camera 21...Display 22...Input interface 24...Processing circuit 3, 4, 5... Sub-PC

Claims

1. a display unit that displays a block representing a device that performs input or output and a block representing a process involving input and output, and further displays a selected block; an operation unit that enables an operation of selecting the blocks on the display unit, connecting a connector indicating the output in one of the selected blocks with a connector indicating the input in another of the selected blocks, and grouping a plurality of the selected blocks; a processing unit that allocates processes indicated by the selected blocks to processing resources in accordance with the connection states between the blocks and the grouping, and then determines an order in which the threads are to be executed, and executes the processes allocated to the processing resources in accordance with the order; A processing device comprising:

2. The processing unit selects one of the selected blocks. When a connector indicating the output in one block is connected to a connector indicating the input in another block, the process indicated by the one block is executed, and then the process indicated by the other block is executed; When the connectors representing the plurality of outputs in one block are connected to the connectors representing the plurality of inputs in a plurality of blocks, the processes represented by the plurality of blocks are executed in parallel. The processing device of claim 1 .

3. The processing unit When each connector indicating a plurality of outputs in one block is connected to each connector indicating an input in multiple blocks, When the shared resources used by the processes indicated by the plurality of blocks overlap, the processes are executed sequentially; When the processes indicated by the multiple blocks do not use shared resources or do not overlap, the processes are executed in parallel. The processing device according to claim 2 .

4. The processing unit When a plurality of the selected blocks are grouped, the processes indicated by the plurality of the blocks are assigned to one thread, and then the processes indicated by the plurality of the blocks assigned to the thread are sequentially executed. The processing device of claim 1 .

5. The device for performing the input includes a camera capable of photographing a subject or a sensor for detecting a physical quantity, the processing unit executes processing based on the image data of the subject acquired from the camera or the physical quantity acquired from the sensor, and causes the display unit to display a result of the processing. The processing device of claim 1 .

6. The device for outputting the information includes an alarm device, a multi-screen display, or a robot arm, The processing unit activates the alarm device, the multi-screen display, or the robot arm when a predetermined condition is satisfied. The processing device of claim 1 .

7. The processing unit executes machine learning processing including face recognition as the processing. The processing device of claim 1 .

8. A processing system comprising one main processing unit and a plurality of sub-processors, The main processing unit a display unit that displays a block representing a device that performs input or output and a block representing a process involving input and output, and further displays a selected block; an operation unit that enables an operation of selecting the blocks on the display unit, connecting a connector indicating the output in one of the selected blocks with a connector indicating the input in another of the selected blocks, and grouping a plurality of the selected blocks; a processing unit that allocates processes indicated by the selected blocks to processing resources in accordance with the connection state between the selected blocks and the grouping, and then determines the order in which the processing resources are to be executed, and instructs the sub-processors to execute the processes allocated to the processing resources in accordance with the order; Equipped with The processing unit Acquire information including the processing performance of the sub-processing device; and further instructing the sub-processing device to execute the process based on the information. Processing system.

9. The information includes data transmission / reception performance between the main processing device and the sub-processing device. The processing system of claim 8 .

10. The information includes a usage fee for the sub-processing device. The processing system of claim 8 .

11. The processing unit The type and amount of data of the processing, the sub-processing device that executed the processing, the processing time, usage rate, and temperature of the sub-processing device are acquired each time; training a machine learning model using the type of processing, the amount of data, and the sub-processing device that executed the processing as explanatory variables, and the processing time, usage rate, and temperature of the sub-processing device as objective variables; By generalizing the machine learning model, a processing time, a usage rate, and a temperature of the sub-processing device are predicted based on the type and amount of data of the processing and the sub-processing device that executes the processing. The processing system of claim 8 .

Images