Control system and control method

Abstract

To track an arbitrary object that moves irregularly and observe the object from any angle of view.SOLUTION: A control system according to the present application includes a reception unit that receives, from a user, a specification of a desired operation when a camera photographs a predetermined object, an extraction unit that extracts characteristics of the predetermined object from an image including the predetermined object, an identification unit that identifies an object having the characteristics as the predetermined object from the image photographed by the camera, and an operation control unit that controls the operation of the camera to perform the desired operation when the predetermined object is identified from the image photographed by the camera on the basis of the specification from the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a control system and a control method. [Background technology]

[0002] A technique for changing the frame rate depending on the magnitude of the movement of a subject is known. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2024-011715 Summary of the Invention [Problem to be solved by the invention]

[0004] However, with conventional technology, when mechanically photographing an object (subject) with a camera, it is possible to track an object that moves regularly by inputting the pattern in advance, but it is difficult to track an object that moves irregularly. Furthermore, not only tracking, but even if you have an image of how you want to capture the object, it is difficult to operate the camera automatically.

[0005] The present application has been made in view of the above, and aims to track any object that moves irregularly and observe the object from any angle of view. [Means for solving the problem]

[0006] The control system of the present application is characterized by comprising: a reception unit that receives from a user a specification of a desired operation when a camera photographs a predetermined object; an extraction unit that extracts characteristics of the predetermined object from an image including the predetermined object; an identification unit that identifies an object having the characteristics as the predetermined object from the image photographed by the camera; and an operation control unit that controls the operation of the camera to perform the desired operation when the predetermined object is identified from the image photographed by the camera based on the specification from the user. [Effects of the Invention]

[0007] According to one aspect of the embodiment, it is possible to track any object that moves irregularly and observe the object from any angle of view. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of a control system according to an embodiment. [Figure 2] FIG. 2 is an explanatory diagram showing an overview of the learning phase. [Figure 3] FIG. 3 is an explanatory diagram showing an overview of the control phase. [Figure 4] Figure 4 is a table showing examples of scenes where this technology can be used. [Figure 5] FIG. 5 is a diagram illustrating an example of the configuration of the control server according to the embodiment. [Figure 6] FIG. 6 is a flowchart illustrating an example of the processing flow of the control system according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] The control system 1 according to the embodiment described below includes a receiving unit 331, an extracting unit 333, an identifying unit 334, and an operation control unit 337. The receiving unit 331 receives from the user a designation of a desired operation to be performed when the camera 10 captures an image of the predetermined target. The extracting unit 333 extracts features of the predetermined target from an image including the target. The identifying unit 334 identifies an object having the feature in the video captured by the camera 10 as the predetermined target. The operation control unit 337 controls the operation of the camera 10 to perform the desired operation when the predetermined target is identified from the video captured by the camera 10 based on the designation from the user.

[0010] The control system 1 according to the embodiment described below further includes a learning unit 332. The learning unit 332 learns images containing a predetermined object, text related to the predetermined object, and features of the predetermined object as learning data, and constructs a predictive AI that, when an image and text containing the predetermined object are input, extracts the features of the predetermined object from the image. The extraction unit 333 uses the predictive AI to extract the features of the predetermined object from the video captured by the camera 10.

[0011] Furthermore, the control system 1 according to the embodiment described below further includes a prompt generation unit 335 and a control information generation unit 336. The prompt generation unit 335 generates a prompt based on a desired action designated by the user and a predetermined target. The control information generation unit 336 inputs the prompt into a generation system AI to generate control information for the camera 10 to perform the desired action. The action control unit 337 controls the action of the camera 10 to perform the desired action based on the control information.

[0012] Furthermore, in the control system 1 according to the embodiment described below, the extraction unit 333 extracts features of a range including a predetermined object from the video captured by the camera 10, the identification unit 334 identifies the range including the predetermined object as a range of interest, and the prompt generation unit 335 generates a prompt based on the user's specification of the desired action and the range of interest.

[0013] In addition, in the control system 1 according to the embodiment described below, the operation control unit 337 controls the operation of the camera 10 when the camera 10 photographs a part being transported on a factory production line as a specified target, with the desired operation being to identify the shape of the part.

[0014] Hereinafter, a control system according to an embodiment will be described with reference to the drawings. The same components in the embodiments are denoted by the same reference numerals, and duplicated descriptions will be omitted.

[0015] FIG. 1 is a diagram illustrating an example of the configuration of a control system according to an embodiment. As illustrated in FIG. 1, the control system 1 includes a camera 10, a repeater 20, a control server 30, and a work terminal 40. The camera 10 is an imaging device with a photographing function. Alternatively, the camera 10 may be a camera-equipped LED light integrated with a lighting fixture. The repeater 20 is a relay device such as a router that relays communications. Alternatively, the repeater 20 may be a base station or the like. The control server 30 is a server device located on a network. The work terminal 40 is a terminal device such as a personal computer (PC) used by a user. Alternatively, the work terminal 40 may be a smart device such as a smartphone or tablet terminal. In reality, the user operating the work terminal 40 may be a worker, operator, work manager, supervisor, or the like.

[0016] The cameras 10 and the repeaters 20, the repeaters 20 and the control server 30, and the control server 30 and the work terminal 40 are connected wirelessly or by wire to enable direct or indirect communication. There may be a plurality of cameras 10 and a plurality of repeaters 20. The control system 1 may be configured without the repeater 20 by directly connecting the cameras 10 and the control server 30. The control server 30 is installed on the cloud 2 and is connected to each device via a network such as a LAN (Local Area Network) or the Internet. The control server 30 may be configured by a plurality of servers on the cloud 2.

[0017] Furthermore, the control server 30 controls the operation of the camera 10 via the repeater 20 (or another communication path). For example, the control server 30 controls the position and attitude, angle of view, zoom, frame rate, etc. of the camera 10. The control server 30 may also control the operation of machines in factory equipment (production equipment). For example, the control server 30 may control the operation of a robot arm or a machine tool different from a robot arm, in addition to the belt conveyor shown in FIG. 1. Furthermore, the control server 30 may control the operation (timing, position, number, etc.) of placing parts to be processed on the belt conveyor. In reality, the camera 10 and the machines in the factory equipment have functions necessary for operation control and may operate independently (under autonomous control) without going through the control server 30.

[0018] In this embodiment, a scene in which equipment is operating in a factory is assumed. In FIG. 1, a camera 10 captures the operation of machinery in the factory equipment. As shown in FIG. 1, the video (video) capturing the scene shows (A) a belt conveyor operating, and (B) parts flowing on the belt conveyor. The belt conveyor is constantly moving, and parts processed in a previous process flow irregularly along it.

[0019] In this case, for example, if a part of the same shape as the part viewed by the arrow in (B) above is flowing by, and you assume that there is a demand to capture it in the center of the angle of view, this embodiment is effective. The method for realizing this is as follows.

[0020] [Learning Phase] First, the control server 30 executes a learning phase for the predictive AI, which is the first AI (Artificial Intelligence). The predictive AI learns the characteristics (feature quantities) of objects captured in the video captured by the camera 10. Alternatively, the predictive AI learns based on learning data such as video and image data prepared in advance. While there is a method for learning using only the objects captured in the video, another method is to input information about the shapes of parts to be manufactured in advance and learn the characteristics of parts similar to those from the video in order to reliably capture the desired objects.

[0021] Possible learning methods include, for example, detecting the iconic features of a part by photographing it from multiple angles and then detecting parts that have those features from the image, or augmenting pre-prepared images by rotating or enlarging them.

[0022] As described above, the control server 30 causes the predictive AI to learn the association between previously input information and information in the video. Whether the learning is sufficient and complete can be determined by setting arbitrary thresholds in advance for the following four indicators (accuracy rate, precision rate, recall rate, and F-measure).

[0023] (1) Correct answer rate This indicates the degree of accuracy with which the features predicted by predictive AI match the shape of the manufactured part that was input in advance.

[0024] (2) Precision rate This indicates the degree to which the shape of the actual manufactured part matches the shape of the manufactured part predicted by the predictive AI in advance.

[0025] (3) Recall rate This shows how much of the actual manufactured parts the predictive AI has been able to predict.

[0026] (4) F-stop Represents the balance between precision and recall.

[0027] Here, an overview of the learning phase will be described with reference to Fig. 2. Fig. 2 is an explanatory diagram showing an overview of the learning phase.

[0028] The control server 30 generates a set of training images, training texts, and features in advance. The training images are, for example, images of the part 1 photographed from various angles. These images may be actual images (camera images) photographed by the camera 10. The training texts are, for example, various texts describing the part 1. These texts may be texts that conceptually describe the product, such as the product name, product model number, shape (e.g., cylindrical), color, etc. The features are generated, for example, by various known feature extraction techniques from the photographed range of the part that is manually input and transferred to the training image.

[0029] The control server 30 trains the prediction AI to extract features when a training image and training text are input. In this way, the control server 30 constructs a prediction AI that extracts features when a camera image and text are input.

[0030] [Control Phase] Once the learning is complete, the control server 30 moves to the control phase, where it inputs what it wants to achieve and controls the operation of the camera 10. This is made possible by utilizing a second AI, generative AI. Note that a GPT (Generative Pre-trained Transformer) may be used as the generative AI. In this embodiment, a factory worker or manager who wants to change the angle of view of the camera 10 inputs to the generative AI how they want to change the angle of view.

[0031] For example, in the case of this embodiment, what is desired to be achieved is given as a text instruction such as "I want to capture a part with the same shape as the cylindrical part in the foreground on the left side moving on the conveyor belt in the image (see Figure 1) in the center of the field of view," or as an instruction using a moving image such as "I want to click on an object using a mouse or the like based on the captured video, and then capture a part with similar features in the center of the field of view." The generative AI that receives this instruction searches for parts with similar features based on the information it has already learned from the image.

[0032] After the control server 30 determines the angle of view from which the camera 10 wants to capture an image, the process moves to a phase in which the camera 10 actually changes the angle of view. When a specified part or the like is recognized as a result output by the generative AI, the moving part of the camera 10 outputs the result in a format that can be operated, for example, by a movable motor, so that the moving part of the camera 10 captures the object as instructed.

[0033] When operating, it is possible to track moving objects by, for example, determining in real time whether the target position is captured in the center of the field of view by matching the pixel at the center of the image with the feature points captured as feature quantities. Also, if the instructions given to the generative AI by the worker or manager are inappropriate and the intended operation is not performed (although it is generally desirable to have as specific instructions as possible), the desired information will be input into the generative AI and the process will be repeated.

[0034] As an effect of this embodiment, if you instruct the generative AI not only to capture the object in the center of the angle of view as shown by the arrow in Figure 1B, but also to capture it perpendicular to the circle, it becomes possible to measure dimensions such as diameter from the captured image using conventional general image analysis technology. This is effective because it can be achieved by moving the camera 10 when the part itself cannot be moved.

[0035] Here, an overview of the control phase will be described with reference to Fig. 3. Fig. 3 is an explanatory diagram showing an overview of the control phase.

[0036] First, the control server 30 inputs the camera image and text into the predictive AI and extracts features. Here, the camera image includes parts 1 and 2. The text is an instruction sentence to enlarge part 1. The predictive AI extracts features of the area in which part 1 is photographed.

[0037] Next, the control server 30 identifies the range in which the part 1 is photographed as a range of interest based on the camera image and the feature amount.

[0038] Next, the control server 30 generates a prompt based on the text, the identified area of ​​interest, and the current camera setting. Note that the control server 30 may obtain the current camera setting from the camera 10. The reason for using the current camera setting is to handle cases where the content of the text is an instruction to change the current camera setting relative to the current setting (e.g., a little further to the right, a little further forward, etc.). If unnecessary, the current camera setting does not need to be used.

[0039] Next, the control server 30 inputs the prompt into the generative AI and obtains control information as an output. Then, the control server 30 controls the camera 10 based on the control information. That is, the camera 10 operates based on the control information.

[0040] [Example prompt] Examples of prompts to be fed into generative AI include the following sentences: " User input is the text entered by the user to operate the camera. Area of ​​interest is the information that indicates the area of ​​the image captured by the camera in which the noteworthy product is captured. Image is the captured image. Camera settings are the current up / down / left / right camera angle and magnification. Create control information to operate the camera to achieve the shooting mode described in the text. The control information should include the up / down / left / right angles and magnification that change the camera's orientation. User Input: XXXXXXX Featured Area:XXXXX Image: Image data Camera settings: XXXXX "

[0041] [Applicable scenes] It can also be used in scenes other than factories, such as the case shown in Figure 4. Figure 4 is a table showing examples of scenes in which it can be used. For example, as shown in Figure 4, when the expected scene is a "factory," the object to be observed is "irregularly arranged parts," and the desired result is "to identify the shape of the parts."

[0042] Also, if the assumed scene is a "wedding ceremony," the objects of observation are the "bride and groom and guests," and what is desired is to "capture happy expressions such as smiles."

[0043] Also, if the assumed scene is a "sports broadcast," the object of observation is "a player who scores a goal or a point," and what is desired is "to zoom in on the player when a goal is scored."

[0044] Also, if the assumed scene is a "concert," the objects of observation are "singers, performers, and audience," and what is desired is to "change the object of emphasis at the desired timing."

[0045] Furthermore, if the assumed scene is "filming a drama," the objects to be observed are "actors and scenery to be emphasized," and what is desired to be achieved is "to change the object to be emphasized at the desired timing."

[0046] [Control server configuration example] An example of the configuration of the control server 30 will be described with reference to Fig. 5. Fig. 5 is a diagram showing an example of the configuration of the control server according to the embodiment. As shown in Fig. 5, the control server 30 includes a communication unit 31, a storage unit 32, and a control unit 33.

[0047] The communication unit 31 is realized by, for example, a network interface card (NIC), etc. The communication unit 31 is also connected to each device so as to be able to communicate with them via a network N (see FIG. 5) such as a LAN or the Internet.

[0048] The storage unit 32 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as an HDD, an SSD, or an optical disk. The storage unit 32 may store captured image data, acquired sensor information, a factory work schedule, etc. In other words, the storage unit 32 can store output data from the camera 10 and factory equipment.

[0049] The control unit 33 is a controller, and is realized by, for example, a central processing unit (CPU), a micro processing unit (MPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or the like executing various programs (corresponding to examples of information processing programs) stored in a storage device inside the control server 30 in a storage area such as a RAM. In the example shown in FIG. 5, the control unit 33 includes a reception unit 331, a learning unit 332, an extraction unit 333, an identification unit 334, a prompt generation unit 335, a control information generation unit 336, and an operation control unit 337.

[0050] The reception unit 331 receives, from the user's work terminal 40 via the communication unit 31, a designation of a desired operation (or a desired state) when the camera 10 photographs a predetermined target. The reception unit 331 also receives, from the user's work terminal 40 via the communication unit 31, images including the predetermined target, text related to the predetermined target, and features of the predetermined target (or feature amounts of a range including the predetermined target), which serve as learning data.

[0051] The learning unit 332 learns images including a predetermined object, text related to the predetermined object, and features of the predetermined object as learning data, and constructs a predictive AI that, when an image and text including the predetermined object are input, extracts the features of the predetermined object from the image. Note that the image including the predetermined object may be a camera image acquired from the camera 10. The learning unit 332 may also learn images including a predetermined object, text related to the predetermined object, and feature amounts of a range in which the predetermined object is included as learning data, and constructs a predictive AI that, when an image and text including the predetermined object are input, extracts feature amounts of a range in which the predetermined object is included from the image.

[0052] The extraction unit 333 extracts features of a predetermined object from an image including the predetermined object. For example, the extraction unit 333 uses predictive AI to extract features of the predetermined object from a video image captured by the camera 10. The extraction unit 333 may also extract feature amounts of a range including the predetermined object from the video image captured by the camera 10.

[0053] The identification unit 334 identifies an object having a characteristic in the video captured by the camera 10 as a predetermined object. The identification unit 334 may also identify a range that includes the predetermined object as a range of interest. The identification unit 334 may be integrated with the extraction unit 333 or may be included in the extraction unit 333.

[0054] The prompt generation unit 335 generates a prompt based on a user's designation of a desired action and a predetermined target. Alternatively, the prompt generation unit 335 may generate a prompt based on a user's designation of a desired action and a focus area. Alternatively, the prompt generation unit 335 may generate a prompt based on a user's designation of a desired action, a focus area, and a current camera setting.

[0055] The control information generator 336 inputs the prompt into the generation system AI to generate control information for the camera 10 to perform the desired operation.

[0056] The operation control unit 337 performs operation control for the camera 10 to perform a desired operation when a predetermined target is identified from the video captured by the camera 10 based on a user instruction. For example, the operation control unit 337 performs operation control for the camera 10 to perform a desired operation based on control information. Furthermore, the operation control unit 337 may change camera settings when performing operation control.

[0057] When the camera 10 photographs a part being transported on a factory production line as a predetermined target, the operation control unit 337 controls the operation of the camera 10 so that the desired operation is to identify the shape of the part.

[0058] Furthermore, when the camera 10 photographs the bride and groom and guests at a wedding as predetermined subjects, the operation control unit 337 controls the operation of the camera 10 so that the desired operation is to capture happy scenes such as smiles.

[0059] In addition, when the camera 10 is shooting a player who scores a goal or a point during a sports broadcast, the operation control unit 337 controls the operation of the camera 10 so that the desired operation is to zoom in on the player during a scoring scene.

[0060] Furthermore, when the camera 10 photographs a singer or audience at a concert as a predetermined target, the operation control unit 337 controls the operation of the camera 10 so that the desired operation is to change the person to be emphasized at a desired timing.

[0061] The operation control unit 337 controls the operation of the camera 10 so that when the camera 10 is shooting a predetermined subject, such as a performer or a scene to be emphasized in a drama shoot, the desired operation is to change the subject to be emphasized at a desired timing.

[0062] [Processing flow] Next, an example of the processing flow of the control system 1 according to this embodiment will be described with reference to Fig. 6. Fig. 6 is a flowchart illustrating an example of the processing flow of the control system according to this embodiment.

[0063] First, the receiving unit 331 of the control server 30 receives, via the communication unit 31, an image including a predetermined target, text related to the predetermined target, and feature amounts of an area including the predetermined target, which serve as learning data, from the user's work terminal 40. Note that the image including the predetermined target may be acquired from the camera 10.

[0064] Next, the learning unit 332 of the control server 30 learns the image containing the specified object, the text related to the specified object, and the feature values ​​of the range containing the specified object as learning data, and constructs a predictive AI that, when an image and text containing the specified object are input, extracts the feature values ​​of the range containing the specified object from the image (step S102).

[0065] Next, the extraction unit 333 of the control server 30 extracts feature amounts of a range including a predetermined object from the video captured by the camera 10 (step S103).

[0066] Next, the specifying unit 334 of the control server 30 specifies, as a range of interest, a range that includes a predetermined object in the video captured by the camera 10 (step S104).

[0067] Next, the receiving unit 331 of the control server 30 receives, via the communication unit 31, from the user's work terminal 40, a designation of a desired operation when the camera 10 captures an image of a predetermined target (step S105).

[0068] Next, the prompt generating unit 335 of the control server 30 generates a prompt based on the user's designation of the desired action, the attention area, and the current camera settings (step S106).

[0069] Next, the control information generator 336 of the control server 30 inputs the prompt into the generation AI to generate control information for the camera 10 to perform a desired operation (step S107).

[0070] Next, the operation control unit 337 of the control server 30 performs operation control based on the control information so that the camera 10 performs a desired operation (step S108).

[0071] Next, the operation control unit 337 of the control server 30 provides the video captured by the camera 10 to the user's work terminal 40 via the communication unit 31, and prompts the user to confirm whether the desired result has been obtained (step S109). If the desired result has not been obtained (step S109; No), the reception unit 331 of the control server 30 receives, from the user's work terminal 40 via the communication unit 31, a designation of a desired action for the camera 10 to take an image of a predetermined target (return to step S105). That is, if the desired result has not been obtained (the action is not as expected), the user may check the video captured by the camera 10, make corrections, and designate the desired action again. In this case, the processes of steps S105 to S109 may be repeated. Alternatively, if the desired result has been obtained (step S109; Yes), the current situation may be maintained, and the series of processes may be terminated.

[0072] As mentioned above, the innovation from the past is to capture the object with a camera and analyze its features using AI. Based on the analyzed features, the generative AI is used to give instructions on how the person wants to observe, and the object is captured at a corresponding angle of view. This makes it possible to perform arbitrary operations such as tracking an arbitrary object that moves irregularly and observing that object from any angle of view.

[0073] Although an embodiment of the present invention has been described, this embodiment is presented as an example and is not intended to limit the scope of the invention. This embodiment can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. This embodiment is included within the scope and spirit of the invention, as well as within the scope of the invention and its equivalents as set forth in the claims. Furthermore, this embodiment can be appropriately combined within the scope of the processing content. [Explanation of symbols]

[0074] 1. Control System 10 Camera 20 Repeater 30 Control Server 31 Communications Department 32 Storage section 33 Control Unit 331 Reception Department 332 Learning Department 333 Extraction part 334 Specific part 335 Prompt Generation Unit 336 Control Information Generation Unit 337 Motion control section 40 Work Station

Claims

1. a reception unit that receives, from a user, a designation of a desired operation when the camera photographs a predetermined object; an extraction unit that extracts features of the predetermined object from an image including the predetermined object; an identification unit that identifies an object having the characteristic in the image captured by the camera as the predetermined object; an operation control unit that controls the camera to perform the desired operation when the predetermined object is identified from the image captured by the camera based on the user's instruction; A control system comprising:

2. a learning unit that learns an image including the predetermined object, text related to the predetermined object, and features of the predetermined object as learning data, and constructs a predictive AI that extracts features of the predetermined object from the image when the image including the predetermined object and the text are input; Furthermore, The extraction unit extracts features of the predetermined object from the image captured by the camera using the predictive AI.

2. The control system of claim 1.

3. a prompt generation unit that generates a prompt based on the desired action designated by the user and the predetermined object; a control information generation unit that inputs the prompt into a generation system AI to generate control information for the camera to perform the desired operation; Furthermore, The operation control unit performs operation control for the camera to perform the desired operation based on the control information.

2. The control system of claim 1.

4. the extraction unit extracts a feature amount of a range including the predetermined object from the video captured by the camera, the identification unit identifies a range including the predetermined object as a range of interest, The prompt generation unit generates a prompt based on the desired action designated by the user and the attention range.

4. The control system of claim 3.

5. The operation control unit controls the operation of the camera when the camera photographs a part being transported on a production line in a factory as the predetermined object, by setting the desired operation to identifying the shape of the part.

2. The control system of claim 1.

6. A control method executed by an information processing device, a receiving step of receiving, from a user, a designation of a desired operation when the camera photographs a predetermined object; an extraction step of extracting features of the predetermined object from an image including the predetermined object; an identifying step of identifying an object having the characteristic in the image captured by the camera as the predetermined object; an operation control step of controlling the camera to perform the desired operation when the predetermined object is identified from the image captured by the camera based on the user's instruction; A control method comprising:

Images