Imaging system, imaging program, and imaging method

Abstract

The purpose of the present invention is to provide an imaging system, an imaging program, and an imaging method that are capable of automatically switching among videos captured by a plurality of cameras and continuously outputting a video at an optimal angle to a viewer. An imaging system 1 is a system that automatically performs switching work of a plurality of imaging devices 10 installed in a stadium, and includes a switching determination unit 20, a control unit 30, and an output unit 40. A switching determination unit 20 selects, from among a plurality of videos captured by the imaging devices 10, an imaging device 10 that outputs an optimal angle video. The control unit 30 instructs the selected imaging device 10 to perform such a camera operation that provides the optimal screen size, and continues to output the video at the optimal angle to the viewer.

Description

Technical Field

[0001] The present invention relates to an imaging system, an imaging program and an imaging method. In particular, the present invention relates to an imaging system, an imaging program and an imaging method capable of automatically switching between videos captured by multiple cameras and continuously outputting videos of the best angle to viewers. Background Art

[0002] In shooting sports broadcasts, etc., a plurality of cameras are arranged at various places in the playing field so as to shoot players or the ball moving on the field from various directions. Then, the operator uses a switcher to switch between the cameras, thereby performing processing such as selecting a video that may be of most interest to the audience according to the game situation and transmitting the video via broadcast waves or recording the video on a recorder.

[0003] In traditional sports program production, the switching work is performed manually by operators, which requires manpower costs. In addition, during the sports broadcasting process, the operator always needs to check the video captured by each camera, so the workload is large, and the operator needs to be familiar with the rules of the sport and the presentation of the broadcast program.

[0004] Therefore, as an invention that can perform switching work (switching of the switcher) without manual operation, a technology is known that determines who is speaking and recognizes what is being said based on voice information, and switches between videos appropriately (for example, see Patent Document 1).

[0005] Prior art literature

[0006] Patent Literature

[0007] Patent Document 1: Japanese Patent Application Publication No. 2005-295431. Summary of the invention

[0008] Problem that the invention aims to solve

[0009] However, the technology disclosed in the above-mentioned patent document 1 performs switching work based on voice information, which is suitable for programs directed based on the speeches of participants (such as talk shows and information programs), but does not consider its application in sports broadcasting, in which the positional relationship of athletes affects the directing of the program.

[0010] In view of the above viewpoints, an object of the present invention is to provide an imaging system, an imaging program, and an imaging method, which are capable of automatically switching between videos captured by multiple cameras and continuously outputting videos of the best angle to the audience.

[0011] Means of solving the problem

[0012] In order to achieve the above-mentioned purpose, the imaging device of the present invention includes: a plurality of imaging devices that image the same subject from different viewpoints at the same time; a switching determination unit that determines the best angle from videos captured by the plurality of imaging devices and selects an imaging device that outputs a video of the best angle; a control unit that instructs the imaging device selected by the switching determination unit to perform at least one of the following control processes: a reduction process of the subject, an enlargement process of the subject, a tracking process of the subject, and a maintenance process of maintaining a previous screen size; and an output unit that outputs the video captured by the imaging device controlled by the control process based on the control unit.

[0013] Herein, for example, in a sports broadcast, by including multiple imaging devices that image the same subject from different viewpoints at the same time, the game situation of the players can be imaged from all viewpoints, and the progress of the game can be conveyed to the audience in an easy-to-understand manner by outputting videos from these different viewpoints.

[0014] Furthermore, by including a switching determination unit that determines the best angle from among the videos captured by the plurality of imaging devices and selects the imaging device that outputs the best angle video, it is possible to automatically determine the video that can most effectively convey the progress of the game to the audience from among the plurality of videos based on the objective determination of the operator rather than the subjective determination. Furthermore, it is not necessary to assign an operator to perform the switching work, so that efficient switching work can be achieved.

[0015] Furthermore, by including a control unit that instructs the imaging device selected by the switching determination unit to perform at least one of the following control processes: a reduction process for a subject, an enlargement process for a subject, a tracking process for a subject, and a maintenance process for maintaining a previous screen size, a screen size change process (a reduction process for a subject, an enlargement process for a subject, or a tracking process for a subject) or a maintenance process according to a subject situation is performed immediately after the switching determination unit switches from a current video to a video of a different viewpoint. Therefore, the most effective video processing can be performed according to the subject situation, and a realistic video can be output to the audience.

[0016] Furthermore, by including an output unit that outputs a video captured by the imaging device controlled by the control processing based on the control unit, an optimal angle video that has undergone a predetermined video processing can be continuously output to the viewer.

[0017] In addition, the switching determination unit includes a detection unit, which takes a plurality of moving objects playing in a stadium as subjects and detects each of the subjects as a rectangle surrounding each of the subjects in each frame image of a video obtained from an imaging device, thereby enabling the moving objects (including people, animals, cars, etc.) playing in the stadium to be detected, and an area around a predetermined range of the detected moving objects to be specified.

[0018] Furthermore, when the switching determination unit has an inter-rectangular distance measurement unit, the inter-rectangular distance measurement unit measures the inter-rectangular distance and determines the optimal angle based on the inter-rectangular distance, which is the distance between the center coordinates of the rectangles of the moving body as the subject detected by the detection unit, for example, in a game such as a combat sport, a video at an angle in which two players appear side by side as much as possible can be determined as the optimal angle. In this way, a video in which one of the players is blocked due to the players overlapping each other is excluded from the optimal angle, and a video in which the entire movements of both players can be caught can be continuously output to the audience.

[0019] In addition, when the detection unit detects at least two athletes playing a game in a stadium and at least one referee making a decision to judge the game between the athletes as subjects, the optimal angle can be selected based on the positional relationship between the athletes and the referee by detecting the athletes playing the game and the referee judging the game as subjects.

[0020] Furthermore, when the switching determination unit determines as the best angle a video in which the distance between rectangles measured by the distance between rectangles measurement unit is the longest among the videos in which the athlete and the referee do not overlap each other or the athlete is in front of the referee, the video in which the athlete is blocked by the presence of the referee is excluded from the best angle, and a video in which the athletes are as side by side as possible can be selected from the remaining videos as the best angle. Therefore, a video that can capture the entire action of the two athletes can be continuously output to the audience in consideration of the position of the referee.

[0021] In addition, when the detection unit has a uniform color determination unit, the uniform color determination unit determines the uniform color of the athlete in the frame image based on the color information, and detects a specific person from the subject based on the determination result of the uniform color determination unit, and pre-registers the uniform color of the athlete or referee. The subject matching the registered uniform color can be easily detected as the athlete or referee as the specific person.

[0022] In addition, when the detection unit has a moving body detection unit, the moving body detection unit identifies a subject by a background subtraction method or a frame subtraction method, and detects the subject identified by the moving body detection unit as a rectangle surrounding the subject. For example, in a sports broadcast, the background included in a frame image obtained from a video during a game is extracted according to a known algorithm, and an athlete or referee as a subject can be easily detected based on the difference.

[0023] When the enlargement process and the reduction process performed by the control unit are configured as follows, the enlargement process or the reduction process is performed according to the ratio of the width of the rectangle of the subject to the width of the screen frame: when the ratio of the width of the rectangle of the subject detected by the detection unit to the width of the specific frame is greater than a value defined by a predetermined first threshold function, the enlargement process is performed; when the ratio is less than a value defined by a predetermined second threshold function, the reduction process is performed. Therefore, after switching to the video selected by the switching determination unit, the video that can grasp the entire subject can be continuously output to the audience immediately.

[0024] In addition, when the tracking processing of the control unit is configured to track at a constant speed or a constant acceleration so that the midpoints of the center coordinates of one of the rectangles connecting the subjects detected by the detection unit and the other rectangle basically coincide with the center of the frame image, tracking processing can be performed so that at least two athletes detected as subjects are always located in the center of the screen frame.

[0025] In order to achieve the above-mentioned purpose, the imaging program of the present invention enables a computer to perform the following steps: receiving multiple videos of the same subject imaged from different viewpoints at the same time; determining the best angle from the multiple videos and selecting an imaging device that outputs the best angle video; instructing the imaging device that outputs the best angle video to perform at least one of the following control processes: a reduction process of the subject, an enlargement process of the subject, a tracking process of the subject, and a maintenance process of maintaining a previous screen size; and, outputting a video captured by the imaging device controlled by the control process.

[0026] Here, for example, in sports broadcasting, by including the step of receiving a plurality of videos of the same subject imaged from different viewpoints at the same time, a plurality of videos of the game situations of the players imaged from all viewpoints can be received.

[0027] Furthermore, by including the steps of determining the best angle from a plurality of videos and selecting an imaging device that outputs the best angle video, the best angle can be determined from a plurality of received videos captured from different viewpoints based on a predetermined determination condition.

[0028] In addition, at least one of the following control processes is performed by the imaging device including an instruction to output the best angle video: a zoom-out process of the subject, an enlargement process of the subject, a tracking process of the subject, and a maintenance process of maintaining the previous screen size. After switching from the current video to a video of a different viewpoint, a screen size change process or a maintenance process is immediately performed according to the situation of the subject. Therefore, the most effective video processing can be performed according to the situation of the subject, and a realistic video can be output to the audience.

[0029] Furthermore, by including the step of outputting the video captured by the imaging device controlled based on the control process, the best angle video subjected to the predetermined video process can be continuously output to the audience.

[0030] In addition, the step of selecting an imaging device that outputs an optimal angle video includes the following steps: using multiple moving objects playing in a stadium as subjects and detecting each of the subjects as a rectangle surrounding each of the subjects in each frame image of the video obtained from the imaging device, so that moving objects (including people, animals, cars, etc.) playing in the stadium can be detected from the subjects, and an area around a predetermined range of the detected moving objects can be specified.

[0031] Furthermore, by including the steps of measuring the distance between rectangles and determining the video having the longest distance between rectangles as the optimal angle, the distance between rectangles being the distance between the center coordinates of the rectangles of the moving body as the subject, for example, in a combat sports competition, a video at an angle in which two players appear side by side as much as possible can be determined as the optimal angle. In this way, a video in which one of the players is blocked due to the players overlapping each other is excluded from the optimal angle, and a video in which the entire movements of both players can be captured can be continuously output to the audience.

[0032] In addition, when the step of selecting an imaging device for outputting an optimal angle video also includes the step of detecting at least two athletes playing a game in a stadium and at least one referee who makes decisions for judging the athletes' game as subjects, the optimal angle can be selected based on the positional relationship between the athletes and the referee by detecting the athletes playing the game and the referee who makes decisions for judging the game as subjects.

[0033] Furthermore, when the step of determining the video with the longest distance between rectangles as the best angle includes the step of determining the video with the longest distance between rectangles in the video where the athlete and the referee do not overlap each other or the video where the athlete is in front of the referee as the best angle, the video where the athlete is blocked by the presence of the referee is excluded from the best angle, and the video where the athletes are as side by side as possible can be selected from the remaining videos as the best angle. Therefore, considering the position of the referee, the video that can capture the entire action of the two athletes can be continuously output to the audience.

[0034] In addition, when the step of detecting each of at least two athletes playing in a stadium as a rectangle surrounding each athlete includes the step of determining the uniform color of the athletes in a frame image based on predetermined color information and the step of performing noise processing on the frame image through morphological transformation, the uniform colors of the athletes are registered in advance and a subject matching the registered uniform color can be easily determined as the athlete as a detection target.

[0035] In addition, when the step of detecting the referee's decision on the athlete's game includes the step of grayscale the frame image and the step of noise processing the frame image through morphological transformation, since the referee's uniform is usually monochrome with white or black as the base color, the referee as the subject can be easily detected from the frame image by grayscale the frame image.

[0036] In order to achieve the above-mentioned purpose, the imaging method of the present invention includes the following steps: receiving multiple videos of the same subject imaged from different viewpoints at the same time by imaging devices arranged at different positions; determining the best angle from the multiple videos and selecting the imaging device that outputs the video of the best angle; instructing the imaging device that outputs the video of the best angle to perform at least one of the following control processes: a reduction process of the subject, an enlargement process of the subject, a tracking process of the subject, and a maintenance process of maintaining the previous screen size; and, outputting the video captured by the imaging device controlled by the control process.

[0037] Through the above steps, a video with the best angle can be selected and output from videos captured from different viewpoints.

[0038] Effects of the Invention

[0039] The imaging system, imaging program and imaging method according to the present invention can automatically switch between videos captured by multiple cameras and continuously output videos with the best angle to the audience. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 is a schematic diagram showing the overall configuration of an imaging system according to an embodiment of the present invention.

[0041] Figure 2 It is a diagram showing the flow of processing in the detection unit.

[0042] Figure 3 Schematic diagram showing the processing steps of moving object detection using the background difference method.

[0043] Figure 4 2 is a diagram showing a processing flow of the inter-rectangle distance measuring unit.

[0044] Figure 5 It is a diagram showing the processing flow in the control unit. DETAILED DESCRIPTION

[0045] Hereinafter, embodiments of the present invention related to an imaging system, an imaging program, and an imaging method will be described with reference to the accompanying drawings, and the embodiments are provided for understanding of the present invention.

[0046] First, refer to Figure 1The overall configuration of the imaging system 1 according to the embodiment of the present invention is described. For example, the imaging system 1 according to the embodiment of the present invention is applied to live commentary broadcasting of combat sports such as boxing as sports broadcasting. The imaging system 1 is mainly composed of a plurality of imaging devices 10, a switching determination unit 20, a control unit 30, and an output unit 40. Among them, the plurality of imaging devices 10 images a subject (a boxer fighting) from different viewpoints, the switching determination unit 20 selects the best angle video from the videos output by the imaging devices 10, the control unit 30 instructs a predetermined image processing to be performed on the video selected by the switching determination unit 20, and the output unit 40 is used to output the video to the outside.

[0047] 1 Imaging equipment

[0048] The imaging device 10 is a digital video camera capable of capturing still images and moving images. Since the imaging device 10 has a known structure, its detailed description is omitted, but the imaging device 10 is mainly composed of a lens, an aperture, a liquid crystal neutral density (ND) filter (transmittance adjustment unit) and an image sensor, and includes an image display unit that displays a captured video or an image storage unit that temporarily stores a captured video.

[0049] The lens includes various types of lenses (such as a zoom lens and a focus lens), and forms an image of the subject light on the exposure surface of the image sensor through these lenses. The aperture is configured to be able to mechanically adjust the amount of subject light by adjusting its aperture. For example, in response to an instruction from a lens driver, the lens and the aperture can control the zoom lens, the focus lens or the aperture position, or control the aperture of the aperture.

[0050] The liquid crystal ND filter is configured to be able to adjust the transmittance of light (adjust density) according to an applied voltage, and adjusts the amount of subject light incident on the image sensor by adjusting the transmittance according to the applied voltage.

[0051] The image sensor is composed of a complementary metal oxide semiconductor (CMOS) sensor, a charge coupled device (CCD) sensor, etc. The image sensor captures an image by exposing the subject light that has passed through the liquid crystal ND filter and entered. Specifically, the image sensor has a plurality of pixels (red (R) pixels, green (G) pixels, and blue (B) pixels), converts the subject light incident on the exposure surface into an electrical signal for each pixel through photoelectric conversion, and outputs the resulting three primary color signals (R, G, B) as analog image data.

[0052] The configuration of the imaging device 10 has been described above, and a plurality of imaging devices 10 are installed around a boxing ring in a stadium where a combat sport (boxing in the embodiment of the present invention) competition is conducted. Specifically, one imaging device is provided on each side, that is, at least four imaging devices 10a, 10b, 10c, and 10d are arranged and installed around a square boxing ring R in a boxing stadium. The four imaging devices 10a, 10b, 10c, and 10d can image boxers fighting on the boxing ring R from different viewpoints.

[0053] Note that the number of imaging devices 10 is not necessarily limited to four, and as long as videos from different viewpoints can be obtained, the number of imaging devices 10 is not particularly limited. Furthermore, in live commentary broadcasts of football, baseball, etc., which are sports broadcasts, in order to obtain videos from different viewpoints in a large stadium, it is necessary to install imaging devices 10 at all locations in the stadium. In this case, at least four or more imaging devices 10 are installed.

[0054] 2 Switch determination unit

[0055] The switching determination unit 20 has a function of selecting an imaging device 10 that captures an optimal angle video from videos of different viewpoints output by the imaging device 10 , and is composed of a detection unit 21 , an inter-rectangle distance measurement unit 22 , a uniform color determination unit 23 , and a moving body detection unit 24 .

[0056] The detection unit 21 has a function of detecting people as subjects (boxers fighting in a stadium and referees making decisions to judge the boxers' matches) from the video captured by the imaging device 10, and performs detection processing on each specific frame of the video output by the imaging device 10. Specifically, one frame image is extracted from every five frames of the video output by the imaging device 10, and person detection is performed by the detection unit 21 on the extracted frame image.

[0057] Here, the detection of a person by the detection unit 21 does not necessarily have to be performed based on frame images of every five frames, and the time for performing the detection process may be changed as necessary.

[0058] Reference Figure 2 The processing flow of the detection unit 21 is described. First, a frame image of a specific frame unit is selected from the video output by the imaging device 10, and moving body (person) detection is performed by the moving body detection unit 24 (S11). The moving body detection unit 24 detects moving bodies (persons include boxers, referees, and spectators) from the video in the frame image.

[0059] At this time, once a moving body is detected, the detection result is temporarily stored in the storage unit. If the moving body detection unit 24 does not detect a moving body, the past detection result temporarily stored in the storage unit is read out and used for subsequent processing. Note that the range of past detection results that can be reused in this case can be set arbitrarily, and in the embodiment of the present invention, it is set to two seconds.

[0060] The moving body detection unit 24 performs moving body detection on the selected frame image based on the background difference method. The background difference method is a known image processing method that extracts the target subject by obtaining the difference between two frame images captured at the same position at different times (for example, now and in the past). In the embodiment of the present invention, based on the known background difference algorithm, the background contained in the frame image G obtained from the video being matched is extracted, and the person as the moving body is identified from the difference.

[0061] Figure 3 is a schematic diagram showing a process of performing moving object detection from a frame image G using a background difference method. Figure 3 As shown, when moving body detection is performed from the frame image G using the background difference method, a binary first mask image M1 is obtained from the frame image G, in which the moving bodies (boxers, referees, and spectators) are represented by white and the other parts are represented by black.

[0062] From the first mask image M1, personnel detection of the boxer and the referee (excluding the audience) is further performed. First, in the detection of the boxer, a color image C in which the white part of the first mask image M1 is colored is generated based on the frame image G, and the color of the uniform worn by the boxer is detected from the obtained color image C (S12).

[0063] In uniform color detection, for example, the colors of wearable uniforms specified in the boxing match rules are pre-registered in the uniform color determination unit 23, the color of the uniform worn by the boxer is extracted from the color image C, and the extracted uniform color is compared with the uniform color registered in the uniform color determination unit 23, and each fighting boxer is identified from the color image C.

[0064] In the color image C, a large amount of noise exists in the boundary portion between the black mask region as the background region and the person as the subject, which makes the range of the subject unclear. Therefore, in order to make the boundary between the subject and the mask region clearer, a known noise processing (such as morphological transformation, filtering processing, or bilateral filtering) is performed to remove the noise existing at the boundary between the mask region and the subject (S13).

[0065] If one of the two fighting boxers can be detected after the noise processing, a binary second mask image M2 is obtained in which the detected boxer is white and the other persons and the background are black. A rectangle surrounding the entire white portion of the second mask image M2 is then defined, and one boxer can be detected from the second mask image M2 as a first rectangle B1. By applying the above-mentioned processing to the other boxer as an opponent, the other boxer is detected from the second mask image M2 as a second rectangle B2 (S14).

[0066] Note that when the fighting boxers are defined as the first rectangle B1 and the second rectangle B2, respectively, a third rectangle B3 surrounding the first rectangle B1 and the second rectangle B2 may also be defined. The third rectangle B3 is used for control processing in the control unit 30 described later.

[0067] After detecting the boxer who is fighting, the detection of the referee is then performed. In many cases, the uniform of the referee basically has a color based on white or black, so in an embodiment of the present invention, the frame image is first grayscaled (S15), the "white" with the highest brightness is identified from the grayscaled image, and the uniform of the referee is detected (S16). Once the referee can be detected, similar to the detection process of the boxer, the noise processing of the boundary between the referee and the background appearing in the frame image is performed (S17), and the definition of the rectangle surrounding the entire referee is completed (S18).

[0068] Here, the referee's uniform is not necessarily identified from the grayscale frame image. For example, similar to the detection of a boxer's uniform, the referee's uniform color specified in the competition rules can be pre-stored in a storage unit, and the uniform color stored in the storage unit can be compared with the color in the frame image to identify the referee's uniform.

[0069] Furthermore, the boxer and the referee do not have to be defined as rectangles in the person detection by the detection unit 21. The definition of a rectangle is an example, and the definition of the detected person can be appropriately selected from known methods.

[0070] The inter-rectangle distance measuring unit 22 has a function of measuring the distance between rectangles defined for each boxer by the detection unit 21. Specifically, the inter-rectangle distance measuring unit 22 defines the center coordinates of the first rectangle B1 and the second rectangle B2 of the two boxers fighting in a plurality of frame images captured by the imaging device 10. Then, the inter-rectangle distance measuring unit 22 measures the distance between the center coordinates and defines the video having the longest inter-rectangle distance as the optimal angle.

[0071] That is, the video with the longest inter-rectangular distance of the boxers is a video in which there is little overlap between the boxers and the boxers appear side by side, and has a composition in which the movements of each boxer can be best seen, and such a video is defined as the best angle in the present invention. Therefore, the imaging device that outputs the video with the longest inter-rectangular distance of the boxers is selected as the imaging device that outputs the best angle video.

[0072] However, when the referee appears in the selected frame image, the relative positional relationship between the referee and the boxer needs to be considered. Therefore, considering the relative positional relationship between the boxer and the referee, the inter-rectangular distance measuring unit 22 selects a video in which the boxer and the referee do not overlap each other or a video in which the boxer is located in front of the referee from the output video, and determines the video with the longest inter-rectangular distance of the boxer among these videos as the optimal angle. This makes it possible to prevent a video in which the boxer's action is blocked by the referee from being erroneously determined as the optimal angle.

[0073] Figure 4 2 is a diagram showing the processing flow of the switching determination unit 20. First, the distance between rectangles of the boxer in the current frame image captured by the imaging device 10 is measured (S21), and the measurement result is stored in the storage unit. Next, it is determined whether the referee appears in front of the boxer (S22). As a result of the determination in S22, if the referee appears in front of the boxer, it is difficult for the audience to confirm the situation of the boxer's fight. Therefore, it is determined that the imaging device 10 needs to be switched, and the process moves to S23.

[0074] On the other hand, if it is determined that the referee is not in front of the boxer according to the determination result in S22, it is determined whether five seconds or more have passed since the previous switch (S24) to further determine whether the imaging device 10 needs to be switched.

[0075] If it is determined in S24 that five seconds or more have passed since the previous switch, it is determined that the imaging device 10 needs to be switched because the video from the same angle has continued for a certain period of time, and the processing moves to S25. On the other hand, if it is within five seconds since the previous switch, the video captured by the current imaging device 10 is maintained, and the processing flow is terminated.

[0076] Note that, as the determination condition in S24, the time that has passed since the previous switching of the video does not have to be set to five seconds, and may be set arbitrarily.

[0077] If more than five seconds have passed since the previous switch in S24, the process moves to S25, and determines whether the inter-rectangular distance of the boxers is within a predetermined range. That is, in S25, if the distance between the boxers is short based on the inter-rectangular distance measured in S21, the video causes the boxers to overlap each other, making it difficult for the audience to understand the fighting situation, so it is determined that the imaging device 10 needs to be switched, and the process moves to S23.

[0078] On the other hand, if the boxers are at a predetermined distance from each other, or if the boxers are fighting at a very close range, the current video is maintained, and the processing flow is terminated. Herein, the reason why the video captured by the current imaging device 10 is maintained when the boxers are fighting at a close range is that the video causes the boxers to overlap each other in all the imaging devices 10, and visibility cannot be expected to be improved by switching. Note that the distance between the boxers used in the determination in S25 can be set arbitrarily.

[0079] If it is determined in S22 and S25 that the imaging device 10 needs to be switched, the process moves to S23. Then, in S23, a specific imaging device 10 is excluded from the switching candidate. Specifically, among the frame images output by the imaging device 10, the following imaging devices 10 are excluded from the switching candidate imaging devices 10: an imaging device 10 that outputs a frame image in which a rectangle of one of the boxers is not detected or a frame image in which both boxers are not detected, and an imaging device 10 that outputs a frame image in which a referee is detected in front of the boxers.

[0080] Then, moving to S26, it is determined whether the distance between the rectangles of the boxers in the current frame image and the switching frame image among the frame images output by the imaging devices 10 (except the imaging devices 10 excluded in S23) is equal to or greater than a specific value (S27). More specifically, it is determined whether the difference between the average value of the distance between the rectangles of the boxers in the past second and the average value of the distance between the rectangles of the boxers in the current second is equal to or greater than a predetermined value, and the imaging device 10 that outputs the frame image with the longest distance between the rectangles of the boxers is selected as the imaging device to be switched next.

[0081] Then, if it is determined in the determination of S27 that the distance between the boxers is equal to or greater than the specific value, switching to the specific imaging device 10 is performed, and if the distance between the boxers is less than the specific value, switching is stopped and the processing flow is terminated.

[0082] 3Control Unit

[0083] Next, we will refer to Figure 5The flowchart of FIG. 3 describes the control process of the control unit 30. If the imaging device 10 outputting the best angle video is selected by the switching determination unit 20 and the imaging device 10 is switched (S31), the control unit 30 has a function of instructing the switched imaging device 10 to perform output control of the video.

[0084] In the output control of the video, the screen size is compared with the rectangle detected in the detection unit 21 in the output video of the switched imaging device 10, and it is determined whether to perform one of the following processes: reduction processing, enlargement processing, tracking processing, and maintenance processing of maintaining the previous screen size (S32).

[0085] First, regarding the reduction processing (S33) and the enlargement processing (S34), the output video of the imaging device 10 before switching is compared with the output video of the imaging device 10 after switching. If the output video after switching shows a larger boxer relative to the output screen size, the reduction processing is performed. Conversely, if the boxer is smaller than the screen size, the enlargement processing is performed and the size of the boxer is adjusted to a size suitable for the screen size.

[0086] When determining the reduction process and the enlargement process, the variable threshold process to be performed is determined based on the width of the rectangle detected by the detection unit 21. Specifically, based on the threshold function defined by the following expressions (1) and (2), if the ratio of the width of the boxer's rectangle to the screen frame of the frame image is less than the value of expression (1) (the threshold function for the reduction process), the reduction process is performed; if the ratio is greater than the value of expression (2) (the threshold function for the enlargement process), the enlargement process is performed. Note that w in expressions (1) and (2) is the width of the boxer's rectangle detected by the detection unit 21.

[0087] 26064x w^(-1.936)+1.1(1)

[0088] 3500.7xw^(-1.465)+1.25(2)

[0089] Herein, the reduction processing and the enlargement processing do not necessarily have to be determined based on the threshold function of the above-mentioned expressions (1) and (2), and the determination criteria may be changed as necessary.

[0090] Next, the tracking process (S35) will be described. First, the center coordinates of the third rectangle B3 surrounding the two boxers in the frame image captured by the imaging device 10 are defined. Then, the offset between the center coordinates and the center position of the frame image is calculated. If the offset is equal to or greater than a predetermined value, it is determined that the boxer is displayed at a position offset from the center position of the frame image, and a pan or tilt tracking process is instructed to the imaging device 10 so that the center coordinates of the third rectangle B3 match the center position of the frame image.

[0091] Herein, in the determination of the tracking process, it is not necessary to calculate the offset between the center coordinates of the third rectangle B3 and the center position of the frame image. For example, the midpoint of the straight line connecting the center coordinates of the first rectangle B1 and the center coordinates of the second rectangle B2 can be defined to calculate the offset between the midpoint and the center position of the frame image.

[0092] In addition, the tracking process is basically constant speed tracking, but constant acceleration tracking of accelerated speed tracking can be performed according to the state of the boxer. For example, in a boxing match, the boxer does not always move in the boxing ring, and sometimes stops moving on the spot to check the opponent. When tracking the movement of the boxer in such a stationary state, the initial movement tracked by the imaging device 10 is controlled with constant acceleration, thereby realizing camera operation without giving the audience a sense of incongruity.

[0093] As described above, if the enlargement process, the reduction process, and the tracking process are performed based on the relationship between the screen size of the frame image and the rectangle detected by the detection unit 21, the screen is updated (S36), and the updated video is output from the output unit 40. On the other hand, if the control of the control unit 30 is not required before and after the switching of the imaging device 10, the previous screen size is maintained unchanged (S37).

[0094] 4 output units

[0095] After the switching determination unit 20 selects the imaging device 10 to be switched and the control unit 30 performs control of the imaging device 10 , the output unit 40 outputs the video captured by the imaging device 10 .

[0096] The imaging system, imaging program, and imaging method according to the embodiments of the present invention have been described above. However, the technical scope of the present invention is not limited to the above embodiments, and various inventions can be configured by appropriate combinations of multiple components disclosed in the above embodiments without departing from the main purpose of the present invention. For example, the present invention can be applied not only to sports broadcasts of combat sports including boxing, but also to various sports broadcasts such as ball games, swimming, car sports, and further horse racing with animals as the main characters. In this case, the detection unit 21 detects the car or animal as the main character of each game as the subject and determines the optimal angle.

[0097] As described above, the imaging system, imaging program, and imaging method according to the present invention can automatically switch between videos captured by multiple cameras and continuously output videos at the best angle to the audience.

[0098] Description of Reference Numerals

[0099] 1: Imaging system

[0100] 10, 10a, 10b, 10c, 10d: Imaging equipment

[0101] 20: Switch to confirm unit

[0102] 21: Detection unit

[0103] 22: Rectangular distance measurement unit

[0104] 23: Uniform color determination unit

[0105] 24: Moving object detection unit

[0106] 30: Control unit

[0107] 40: Output unit

[0108] B1: First rectangle

[0109] B2: Second rectangle

[0110] B3: The third rectangle

[0111] C: Color image

[0112] G: frame image

[0113] M1: first mask image

[0114] M2: Second mask image

[0115] R: Boxing Ring

Claims

1. An imaging system, comprising: multiple imaging devices that image the same subject from different viewpoints at the same time; a switching determination unit that determines an optimal angle from among the videos captured by the plurality of imaging devices, and selects an imaging device that outputs the video of the optimal angle; a control unit that instructs the imaging device selected by the switching determination unit to perform at least one of the following control processes: a reduction process for the subject, an enlargement process for the subject, a tracking process for the subject, and a maintenance process for maintaining a previous screen size; as well as An output unit that outputs a video captured by the imaging device controlled by the control process based on the control unit.

2. The imaging system according to claim 1, wherein: The switching determination unit has: a detection unit that uses a plurality of moving bodies playing in a stadium as the subjects and detects each of the subjects as a rectangle surrounding each of the subjects from each frame image of the video obtained by the imaging device; and an inter-rectangle distance measuring unit that measures an inter-rectangle distance that is a distance between center coordinates of a rectangle that is a moving body of a subject detected by the detecting unit, and The optimal angle is determined according to the distance between the rectangles.

3. The imaging system according to claim 2, wherein: The detection unit detects at least two athletes playing in a stadium and at least one referee making a decision for judging the game of the athletes as the subjects, and The switching determination unit determines, as the optimal angle, a video in which the distance between the rectangles is the longest from among the videos in which the player and the referee do not overlap each other or the videos in which the player is located in front of the referee.

4. The imaging system according to claim 3, wherein: The detection unit has a uniform color determination unit that determines the uniform color of the player in the frame image based on color information, and A specific player is detected from the subject based on the determination result of the uniform color determination unit.

5. The imaging system according to claim 2 or 3, wherein: The detection unit has a moving body detection unit that recognizes a subject by a background difference method or a frame difference method, and A subject recognized by the moving body detection unit as a rectangle surrounding the subject is detected.

6. The imaging system according to claim 2, wherein: The enlargement processing and reduction processing performed by the control unit are configured as follows: when the ratio of the width of the rectangle of the subject detected by the detection unit to the width of the specific frame is greater than a value defined by a predetermined first threshold function, the enlargement processing is performed, and when the ratio is less than a value defined by a predetermined second threshold function, the reduction processing is performed.

7. The imaging system according to claim 2, wherein: The tracking processing performed by the control unit is configured to perform tracking at a constant speed or constant acceleration so that the midpoints of the respective center coordinates of one rectangle and the other rectangle connecting the subject detected by the detection unit substantially coincide with the center of the frame image.

8. An imaging program for causing a computer to execute: The step of receiving multiple videos of the same subject imaged from different viewpoints at the same time; The step of determining an optimal angle from the plurality of videos and selecting an imaging device that outputs a video of the optimal angle; A step of instructing an imaging device outputting the optimal angle video to perform at least one of the following control processes: a reduction process of the subject, an enlargement process of the subject, a tracking process of the subject, and a maintenance process of maintaining a previous screen size; as well as The step of outputting a video captured by the imaging device controlled based on the control process.

9. The imaging procedure of claim 8, wherein: The step of selecting an imaging device for outputting the optimal angle video comprises: a step of using a plurality of moving bodies playing in a stadium as subjects and detecting each of the subjects as a rectangle surrounding each of the subjects from each frame image of a video obtained by the imaging device; a step of measuring an inter-rectangular distance, the inter-rectangular distance being a distance between center coordinates of a rectangle of a moving body as the subject; and The step of determining the optimal angle according to the distance between the rectangles.

10. The imaging procedure of claim 9, wherein: The step of selecting the imaging device that outputs the optimal angle video further includes the step of detecting at least two athletes playing in a stadium and at least one referee who makes a decision for judging the athletes' game as the subjects, and The step of determining the video with the longest distance between rectangles as the optimal angle includes: the step of determining the video with the longest distance between rectangles as the optimal angle from the video in which the athlete and the referee do not overlap with each other or the video in which the athlete is located in front of the referee.

11. The imaging procedure of claim 10, wherein: The step of detecting at least two athletes playing in a stadium comprises: The step of determining the uniform color of the player in the frame image based on predetermined color information; and The step of performing noise processing on the frame image by morphological transformation.

12. The imaging procedure of claim 10, wherein: The step of detecting a referee who makes a decision to judge the match of the athlete comprises: A step of graying the frame image; and The step of performing noise processing on the frame image by morphological transformation.

13. An imaging method comprising: The step of receiving a plurality of videos of the same subject imaged from different viewpoints at the same time by imaging devices arranged at different locations; The step of determining an optimal angle from the plurality of videos and selecting an imaging device that outputs a video of the optimal angle; A step of instructing an imaging device outputting the optimal angle video to perform at least one of the following control processes: a reduction process of the subject, an enlargement process of the subject, a tracking process of the subject, and a maintenance process of maintaining a previous screen size; as well as The step of outputting a video captured by the imaging device controlled based on the control process.

Images