Game record generation device and program

Abstract

Generates timecoded game record data from shogi (Japanese chess) match footage. [Solution] The board and piece discrimination unit 10 of the game record generation device 1 uses a board learning model M1 to discriminate the board and piece stand from the game video A, and uses a piece learning model M2 to discriminate the pieces from the video of the board and piece stand, and generates a piece discrimination result. The intermediate table generation unit 11 generates an intermediate table T consisting of the number of moves, board information, and piece stand information based on the piece discrimination result. The game record generation unit 12 generates the move for the latest number of moves based on the board information and piece stand information for the latest number of moves and the move before the latest number of moves included in the intermediate table T, and generates game record data D1 consisting of the number of moves and the move. The game record generation unit 12 also reads the time from the clock 14 to generate a time code, and generates timecoded game record data D2 consisting of the number of moves, the move, and the time code.

Description

Technical Field

[0001] The present invention relates to a game score generation device and a program that utilize object recognition technology by AI to generate shogi game score data from a video of a game board.

Background Art

[0002] Conventionally, shogi game programs have been broadcast, and among them, there are long-running programs that have been broadcast for many years. A large number of game program videos are held as valuable assets in archives. In these shogi game programs, in order to grasp the change in the placement of pieces for each move or analyze the tactics of the players, it is necessary to generate a game score. A game score is a record of the moves made by each player in order with respect to the placement of pieces for each move.

[0003] Since the game scores were generated manually during the recording of shogi game programs, it was inefficient in terms of cost and not fully utilized effectively.

[0004] In order to solve such problems, technologies for automatically generating game score data from shogi game videos have been disclosed (see, for example, Patent Documents 1 and 2).

[0005] The technology described in Patent Document 1 extracts regions where the state of the presence or absence of pieces has changed from the game video, and identifies the type and position of the changed pieces, thereby generating game score data for each move. As a result, game score data can be automatically generated from the video.

[0006] Also, the technology described in Patent Document 2 extracts invalid regions where the shogi board is hidden by obstacles from the game video, extracts regions where the state of the presence or absence of pieces has changed from regions other than the invalid regions, and identifies the type and position of the changed pieces, thereby generating game score data for each move. As a result, not only can game score data be automatically generated from the video, but also when the game score changes in a state where there are obstacles such as the players' moves on the shogi board, the reading errors of the game score changes can be reduced.

Prior Art Documents

[0007] [Patent Document 1] Japanese Patent Application Publication No. 9-220312 [Patent Document 2] Japanese Patent Application Publication No. 9-220311 [Overview of the project] [Problems that the invention aims to solve]

[0008] In order to effectively utilize archives of shogi game footage, not only the game footage but also game record data showing the position of the pieces after each move is necessary, and game record data can be automatically generated using the technologies described in the aforementioned Patent Documents 1 and 2.

[0009] However, the technologies described in Patent Documents 1 and 2 only generate game record data that reflects the changed types and positions of pieces. Therefore, there was a problem in that the game record data generated by the technologies described in Patent Documents 1 and 2 was insufficient for effectively utilizing archives of game videos.

[0010] Furthermore, the footage of shogi matches shown in broadcast programs includes the shogi board used for commentary on actual matches (the board used for large-screen commentary). The arrangement of pieces on the board used for large-screen commentary reflects the result of the commentator freely moving the pieces to explain the actual match, and therefore differs from the arrangement of pieces in the actual match.

[0011] The technologies described in the aforementioned Patent Documents 1 and 2 perform piece detection not only on the board actually used in a game, but also on the board used for large-screen commentary. Furthermore, when identifying the shogi board and pieces from the game video, threshold processing using binarization and matching processing using templates are performed, so the accuracy of identifying the shogi board and pieces cannot be said to be sufficiently high. For this reason, the technologies described in Patent Documents 1 and 2 have the problem that the accuracy of the generated game record data is not always high.

[0012] Thus, there was a need to generate game record data that could effectively utilize archived shogi match videos. Furthermore, the generated game record data needed to be highly accurate.

[0013] Therefore, the present invention has been made to solve the above-mentioned problems, and its objective is to provide a game record generation device and program that generates game record data with time codes from game footage of shogi programs. [Means for solving the problem]

[0014] To solve the above problem, the game record generation device of claim 1 is a game record generation device that generates a game record based on video footage of a shogi match, comprising: a board / piece discrimination unit that uses a predetermined board learning model to discriminate a shogi board and a piece stand from the video footage of the match, and uses a predetermined piece learning model to discriminate pieces from the video footage of the shogi board and the piece stand, and a unit that, based on the type and position of the pieces included in the video footage when the total number of pieces discriminated by the board / piece discrimination unit is 40, generates the number of moves, board information indicating the type and position of the pieces on the shogi board, and piece stand information indicating the type and number of pieces on the piece stand. The system is characterized by comprising: an intermediate table generation unit that generates data for sets and writes the data for sets to an intermediate table, thereby generating an intermediate table consisting of data for multiple sets written in the order of the number of moves; and a game record generation unit that generates moves based on the intermediate table generated by the intermediate table generation unit, generates a time code for the elapsed time from the start time of the game video to the time the moves were generated using the internal clock of the game record generation device, and generates timecoded game record data consisting of the number of moves, the moves, and the time code.

[0015] Furthermore, the game record generation device of claim 2 is characterized in that, in the game record generation device of claim 1, the predetermined board learning model is trained to distinguish from the game video the shogi board and piece stand used in the game, as well as the shogi board and piece stand displayed in the wipe, and the shogi board used for the large-board commentary is not distinguished, and the board and piece discrimination unit uses the predetermined board learning model to distinguish from the game video the shogi board and piece stand used in the game, as well as the shogi board and piece stand displayed in the wipe.

[0016] Furthermore, the game record generation device of claim 3 is characterized in that, in the game record generation device of claim 1, the intermediate table generation unit writes the data of the pair when it generates the data of the pair to the intermediate table as temporary data, and thereafter, when new pair data is generated, if the data of the new pair differs from the temporary data, it overwrites the temporary data in the intermediate table with the data of the new pair as temporary data, and if a predetermined time has elapsed without the temporary data included in the intermediate table changing, it determines the temporary data as regular data and generates the intermediate table.

[0017] Furthermore, the game record generation device of claim 4 is characterized in that, in the game record generation device of claim 1, the game record generation unit generates the moves based on the difference between the board information and piece stand information included in the latest set data included in the intermediate table and the board information and piece stand information included in the set data one step prior to the latest.

[0018] Further, the game record generation device according to claim 5 is the game record generation device according to claim 1, wherein the data of the latest set included in the intermediate table is used as the first data, and the data of the set one move before the latest is used as the second data. The game record generation unit calculates the total number of pieces moved when changing from the number of moves of the second data to the number of moves of the first data based on the first data and the second data. When the total number is 1, it is determined that the change from the number of moves of the second data to the number of moves of the first data is continuous, generates the move instruction, generates the game record data with time code, and when the total number is 2 or more, it is determined that the change is discontinuous, generates a move instruction indicating that the game record is skipped, and generates the game record data with time code.

[0019] Furthermore, the program according to claim 6 causes a computer to function as the game record generation device according to any one of claims 1 to 5.

Advantages of the Invention

[0020] As described above, according to the present invention, game record data with time code can be generated from the game video of a shogi program.

Brief Description of the Drawings

[0021] [Figure 1] It is a block diagram showing a configuration example of a game record generation device according to an embodiment of the present invention. [Figure 2] It is a flowchart showing a processing example of the game record generation device according to an embodiment of the present invention. [Figure 3] It is a flowchart showing a processing example of the board and piece discrimination unit. [Figure 4] It is a flowchart showing a processing example of the intermediate table generation unit. [Figure 5] It is a flowchart showing a processing example of the game record generation unit. [Figure 6] It is a flowchart showing a processing example of the discontinuous move instruction determination process (step S503) by the game record generation unit. [Figure 7]This flowchart shows an example of how the game record embedding section is processed. [Figure 8] This figure shows examples of boards and piece stands that can be identified by the board and piece identification unit. [Figure 9] This figure shows an example of an intermediate table T. [Figure 10] This figure shows an example of game record data D1. [Figure 11] This figure shows an example of game record data D2 with timecodes. [Figure 12] This figure shows an example of the screen for embedded game record video B. [Figure 13] This figure shows an example of a screen displayed by the playback unit. [Modes for carrying out the invention]

[0022] The embodiments for carrying out the present invention will be described in detail below with reference to the drawings. The game record generation device according to an embodiment of the present invention uses AI-based object recognition technology to identify the pieces on the shogi board and the pieces on the piece stand from a shogi game video A, and generates an intermediate table T consisting of the type and position of the pieces on the shogi board and piece stand for each move. The game record generation device then generates moves based on the intermediate table T, generates game record data D1 consisting of the number of moves and moves, and generates timecoded game record data D2 with a timecode added to this, as metadata for the game video A.

[0023] This allows for the generation of timecoded game record data D2 from shogi match video A. Therefore, there is no need to manually generate timecoded game record data D2, which can reduce labor during the recording of shogi programs. Furthermore, timecoded game record data D2 can be used to clearly indicate, for example, the time spent on each move by the shogi players, allowing for the effective use of the game record data.

[0024] [Game record generation device] First, a game record generation device according to an embodiment of the present invention will be described. Figure 1 is a block diagram showing an example of the configuration of a game record generation device according to an embodiment of the present invention, and Figure 2 is a flowchart showing an example of processing by a game record generation device according to an embodiment of the present invention.

[0025] This game record generation device 1 comprises a board / piece discrimination unit 10, an intermediate table generation unit 11, a game record generation unit 12, a game record embedding unit 13, a clock (internal clock) 14, a playback unit 15, and a storage unit 20. The storage unit 20 stores a pre-learned board learning model M1 and a learning model M2 for pieces, and stores the intermediate table T, game record data D1, game record data D2 with timecode, and game record embedded match video B generated by the game record generation device 1.

[0026] The game record generation device 1 receives a video of a shogi game A as input (step S201). The board and piece discrimination unit 10 uses the board learning model M1 stored in the memory unit 20 to discriminate the shogi board (hereinafter referred to as "board") and piece stand from the game video A, and uses the piece learning model M2 stored in the memory unit 20 to discriminate the pieces from the video of the board and piece stand (step S202).

[0027] The board and piece identification unit 10 generates piece identification results indicating the type and position of each piece held by the first player and the second player in the game video A (step S203). The board and piece identification unit 10 then outputs the piece identification results to the intermediate table generation unit 11 and the game record embedding unit 13. Details of the board and piece identification unit 10 will be described later.

[0028] The intermediate table generation unit 11 receives the piece discrimination result from the board / piece discrimination unit 10, generates an intermediate table T based on the piece discrimination result, and stores it in the storage unit 20 (step S204).

[0029] The intermediate table T consists of information about the game state at each stage (per move) as the game progresses with the first and second players taking turns moving their pieces. Specifically, as shown in Figure 9 below, it consists of the number of moves, board information, and piece stand information. Details of the intermediate table generation unit 11 and the intermediate table T will be described later.

[0030] The game record generator 1 moves from step S204 and uses the intermediate table T to determine whether the game has ended (whether it is the final game) (step S205). If it determines that it is not the final game (step S205:N), it moves to step S206. On the other hand, if the game record generator 1 determines in step S205 that it is the final game (step S205:Y), it determines that it has entered the post-game analysis phase and terminates the process.

[0031] Specifically, the game record generator 1 determines the number of pieces that have changed (moved) based on the board information and piece stand information for the most recent move and the move immediately preceding the most recent move, which are included in the intermediate table T. The game record generator 1 then compares the number of changed pieces with a pre-set threshold Z, and determines that the game is not over if the number of changed pieces is less than or equal to the threshold Z. On the other hand, the game record generator 1 determines that the game is over if the number of changed pieces is greater than the threshold Z.

[0032] Thus, using the intermediate table T, if the difference between the position in the most recent move and the position in the move before the most recent move is significant (i.e., if a large number of pieces have changed), it is determined that the game has ended, and the position in the most recent move is determined to be the position in the post-game analysis.

[0033] Furthermore, the pre-set threshold Z (number of pieces moved) used to determine the end of the game is set to be greater than the number of piece movements when a move is determined to be discontinuous, as shown in step S604 of Figure 6, which will be described later.

[0034] The game record generation unit 12 moves from step S205(N) and reads the intermediate table T from the storage unit 20. Then, using the intermediate table T, the game record generation unit 12 generates a move for the latest move, indicating the position of the piece, etc., based on the board information and piece stand information for the latest move and the move immediately preceding the latest move contained in the intermediate table T (step S206). The game record generation unit 12 outputs the move to the game record embedding unit 13.

[0035] The game record generation unit 12 generates game record data D1 using the moves generated in step S206, reads the time from the clock 14, generates game record data D2 with timecode using the moves and time, and stores these in the storage unit 20 (step S207).

[0036] The game record data D1 consists of the number of moves and the move(s), as shown in Figure 10, which will be described later. The game record data D2 with timecodes consists of the number of moves, the move(s), and the timecode, as shown in Figure 11, which will be described later. Details of the game record generation unit 12, game record data D1, and game record data D2 with timecodes will be described later.

[0037] The game record embedding unit 13 receives the game video A as input, the piece identification result from the board / piece identification unit 10, and the moves from the game record generation unit 12. Based on the piece identification result and moves, the game record embedding unit 13 generates a game video with embedded game record B from the game video A and stores it in the storage unit 20 (step S208). Details of the game record embedding unit 13 and the game video with embedded game record B will be described later.

[0038] Furthermore, although not shown in the flowchart in Figure 2, the playback unit 15 displays the game video A, the timecoded game record data D2, and the changes in the AI ​​win rate (win rate calculated by the AI) for each move on the screen. Following user operations, it plays and pauses the game video A, and displays the move number, timecode, and move corresponding to the playback position of the game video A, as well as the AI ​​win rate, on the screen. Examples of screens displayed by the playback unit 15 will be described later.

[0039] <Board / piece identification unit 10> Next, we will explain in detail the board / piece discrimination unit 10 shown in Figure 1. Figure 3 is a flowchart showing an example of processing by the board / piece discrimination unit 10.

[0040] The board and piece discrimination unit 10 receives a video of a shogi game A as input (step S301) and reads the board learning model M1 from the memory unit 20. Then, the board and piece discrimination unit 10 uses the board learning model M1 to distinguish the board and piece stand from the images of the game video A (step S302).

[0041] The board learning model M1 takes an image of the game video A as input data and outputs information such as whether the image contains the board to be processed, whether the image contains the piece stand, and information about the positions of the board and piece stand as output data. The boards to be processed include the board used in the game and the board displayed in the wipe. The board used for the large-screen commentary is not included in the boards to be processed. In other words, the board learning model M1 is trained to distinguish between the board used in the game and the board displayed in the wipe as boards to be processed, but is trained not to distinguish between the board used for the large-screen commentary.

[0042] Figure 8 shows an example of a board and piece stand identified by the board and piece identification unit 10. Figure 8(1) is an example of an image including a board and piece stand used in a game, and the board and piece stand are identified by the board and piece identification unit 10. The lower right of this image includes the piece stand on which the first player's pieces are placed, and the upper left includes the piece stand on which the second player's pieces are placed.

[0043] Figure 8(2) is an example of an image including the board and piece stand displayed in the wipe (see the lower right portion), and the board and piece stand are also identified by the board / piece identification unit 10.

[0044] Figure 8(3) is an example of an image that includes a board used for large-board commentary, and this board is not identified by the board / piece discrimination unit 10 (it is excluded from the boards to be processed). In other words, since the board used for large-board commentary is not a board to be processed by the game record generation device 1, when the board learning model M1 takes the image shown in Figure 8(3) as input data, it generates and outputs information as output data indicating that the image does not contain a board to be processed.

[0045] Thus, the board / piece discrimination unit 10 does not distinguish the board used for the large-board commentary and excludes it from the boards to be processed, making it possible to ignore the movement of pieces by the commentator on the board used for the large-board commentary. In other words, no processing is performed for piece movements by the commentator, and the intermediate table generation unit 11, game record generation unit 12, and game record embedding unit 13 are processed only for piece movements by the players, making it possible to generate a highly accurate intermediate table T, game record data D1, game record data D2 with timecode, and game record embedded match video B.

[0046] Returning to Figure 3, the board / piece discrimination unit 10 reads the piece learning model M2 from the memory unit 20 and uses the piece learning model M2 to distinguish between pieces placed on the board squares and pieces placed on the piece stand from the images of the board and piece stand determined in step S302 (step S303). The board / piece discrimination unit 10 then generates the piece discrimination result and outputs it to the intermediate table generation unit 11 and the game record embedding unit 13 (step S304).

[0047] The chess piece learning model M2 takes an image including the board and piece stand as input data, and outputs information indicating the type and position of pieces placed on each square of the board, as well as the type and number of pieces placed on the piece stand, for both the first and second players.

[0048] The piece identification result is information indicating the type and position of each piece held by the first player and the second player in the game video A. The pieces held by the players are the pieces placed on the squares of the board and the pieces placed on the piece stand (held pieces). The piece identification result is also information regarding the images of the board and piece stand identified from the images of the game video A by the board and piece identification unit 10.

[0049] This allows AI-based object recognition technology using the board learning model M1 and the piece learning model M2 to generate piece identification results that show the type and position of the pieces for the first and second players from the "board capture" footage shown in Figures 8(1) and (2), which frequently appear in the program of game video A.

[0050] Furthermore, for the board learning model M1 and the piece learning model M2, models using an AI object detection algorithm called YORO (You Only Look Once) are used. The YORO object detection algorithm is provided as open-source software (OSS). The board learning model M1 is trained by labeling the board and piece stand to be processed on bounding boxes, and the piece learning model M2 is trained by labeling the pieces on bounding boxes.

[0051] <Intermediate Table Generation Unit 11> Next, we will describe the details of the intermediate table generation unit 11 shown in Figure 1. Figure 4 is a flowchart showing an example of processing by the intermediate table generation unit 11.

[0052] The intermediate table generation unit 11 receives the piece discrimination results from the board / piece discrimination unit 10 (step S401). Then, based on the piece discrimination results, the intermediate table generation unit 11 generates the number of moves, board information, and piece stand information as temporary data (step S402). The board information and piece stand information will be described later.

[0053] The intermediate table generation unit 11 determines whether or not the temporary data (number of moves, board information, and piece stand information) generated in step S402 is present in the last row (the row with the most recent number of moves) of the intermediate table T described later (step S403).

[0054] Figure 9 shows an example of an intermediate table T. This intermediate table T is composed of multiple sets of data, one for each move (one for each finger), where each set of data consists of various pieces of information, including the number of moves, board information, and piece stand information. The number of moves is a numerical value that starts from 0 and increments by 1, with the move in the bottom row being the most recent move. The board information indicates the type and position of the pieces placed on the board during that move, and the piece stand information indicates the type and number of pieces placed on the piece stand during that move.

[0055] For example, if the number of moves is 2, the board information indicates that for Black, a piece of the type "Lance" is placed at square "1-1" (file 1, rank 1), a piece of the type "Knight" is placed at square "2-1", ... and for White, ... a piece of the type "Lance" is placed at square "9-9".

[0056] Returning to Figure 4, if the intermediate table generation unit 11 determines in step S403 that there is no temporary data in the last row of the intermediate table T (step S403:N), it determines that there is regular data in the last row, which will be determined in step S409, as described later. The intermediate table generation unit 11 then adds (writes) the temporary data generated in step S402 to the intermediate table T (step S404) and proceeds to step S401. As a result, the temporary data generated in step S402 is written to the new last row of the intermediate table T. Then, the processing in steps S401 to S403 is performed.

[0057] Although not shown in Figure 9, a flag indicating whether or not the data is temporary is added to the last row of the intermediate table T. If the flag indicates that the data is not temporary, it means that the data in the last row is regular data. In step S404, a flag indicating that the data added to the last row of the intermediate table T is temporary is added to that last row, and in step S409, described later, a flag indicating that the data in that last row is not temporary (i.e., regular data) is added to that last row.

[0058] If the intermediate table generation unit 11 determines in step S403 that there is temporary data in the last row of the intermediate table T (step S403:Y), it compares the temporary data generated in step S402 with the temporary data in the last row of the intermediate table T.

[0059] If the two sets of temporary data are different, the intermediate table generation unit 11 overwrites the last row of the intermediate table T with the temporary data generated in step S402 (step S405) and proceeds to step S406. On the other hand, if the two sets of temporary data are the same, the intermediate table generation unit 11 does not perform the overwrite process described above and proceeds to step S406.

[0060] The intermediate table generation unit 11 moves from step S405 to determine whether a predetermined time has elapsed with the same temporary data in the last row of the intermediate table T, that is, whether the temporary data in the last row has not changed for a predetermined time (whether the temporary data is the same for a predetermined time) (step S406). The predetermined time is, for example, 1 second (the time it takes for 30 frames to elapse when the frame rate is 30fps).

[0061] If the intermediate table generation unit 11 determines in step S406 that the provisional data has not remained the same for a predetermined time (i.e., the provisional data has changed) (step S406:N), it proceeds to step S401. Then, the processing in steps S401 to S403, S405, and S406 is performed.

[0062] On the other hand, in step S406, if the intermediate table generation unit 11 determines that the provisional data has remained the same for a predetermined time (the provisional data has not changed) (step S406: Y), it calculates the total number of pieces held by the first player and the second player based on the piece identification result entered in step S401. Then, the intermediate table generation unit 11 determines whether the total number of pieces is 40 or not (step S407). Here, the total number of pieces held by the first player and the second player is 40.

[0063] If the intermediate table generation unit 11 determines in step S407 that the total number of pieces is not 40 (step S407:N), it proceeds to step S401. Then, the processes in steps S401 to S403 and S405 to S407 are performed.

[0064] For example, in the case of image A of a game where the players' hands have been placed on the board and obscured some pieces, the total number of pieces determined from the piece identification result will be less than 40. In this case, the process moves from step S407(N) to step S401, and therefore, in step S408, which will be described later, it will not be determined that the image is from after a move has been made.

[0065] In other words, for the image of game video A where the players' hands are placed on the board and the pieces are hidden, temporary data is written to the bottom row of intermediate table T, but it is never finalized as regular data, and game record data D1 and game record data D2 with timecode are not generated.

[0066] Furthermore, for example, in the case of image A of the game video at the moment a piece is captured from the opponent, if the piece is hidden by the player's fingers, the total number of pieces will be less than 40, as described above, and therefore it will not be determined to be video after a move in step S408, which will be described later. If the piece is not hidden by the player's fingers, the piece captured from the opponent moves from the board to the piece stand, so new temporary data is generated in step S402, and then, after going through steps S406 and S407, it will be determined to be video after a move in step S408, which will be described later.

[0067] In other words, for the image of game video A at the moment when an opponent's piece is captured, temporary data is written to the bottom row of intermediate table T, similar to the image of game video A where the players' hands are on the board and obscuring the piece, but it is never finalized as regular data, and game record data D1 and game record data D2 with timecode are not generated.

[0068] On the other hand, if the intermediate table generation unit 11 determines in step S407 that the total number of pieces is 40 (step S407:Y), it determines that the game video A (image) corresponding to the piece identification result in step S401 is a video (image) after a move has been made (step S408). In other words, the intermediate table generation unit 11 determines that the image of the game video A is an image taken after a move has been made by either the first or second player (after the move has been determined).

[0069] The intermediate table generation unit 11 confirms the temporary data in the last row written to the intermediate table T as the regular data (step S409). Then, the intermediate table generation unit 11 stores the new intermediate table T, which includes the regular data in the last row, in the storage unit 20 (step S410).

[0070] Thus, game video A including the board used for large-screen commentary, game video A where players' hands are placed on the board and pieces are hidden, and game video A at the moment of capturing an opponent's piece are excluded from processing. This allows for the generation of a highly accurate intermediate table T, and as a result, highly accurate game record data D1, game record data D2 with time codes, and game video B with embedded game records can be generated.

[0071] <Game record generation section 12> Next, we will explain the details of the game record generation unit 12 shown in Figure 1. Figure 5 is a flowchart showing an example of processing by the game record generation unit 12.

[0072] The game record generation unit 12 reads the intermediate table T from the memory unit 20 (step S501). Then, the game record generation unit 12 extracts the move count, board information, and piece stand information of the last row (most recent) from the intermediate table T, as well as the move count, board information, and piece stand information of the row immediately preceding the last row (one row before the most recent) (step S502).

[0073] The game record generation unit 12 performs a move discontinuity determination process to determine whether the moves are discontinuous or not, based on the board information and piece stand information for the number of moves in the last row and the move immediately preceding the last row (step S503).

[0074] A discontinuous sequence of moves indicates that the board and piece stand information corresponding to consecutive moves represents two or more piece movements. Normally, one piece movement occurs with each move made by either the first or second player, and an intermediate table T is generated that reflects this. Therefore, only one piece changes with each change in the number of moves.

[0075] However, the game video A includes the board used for the large-screen commentary, and it is possible that while the commentator is giving their commentary, moves are being made in the background and two or more pieces may be moved. In this case, an intermediate table T is generated that corresponds to two or more piece movements in a series of moves.

[0076] Figure 6 is a flowchart showing an example of the move discontinuity detection process by the game record generation unit 12 (step S503). The game record generation unit 12 determines the number of moves (number of moves) based on the board information and piece stand information for the number of moves in the last row and the move immediately preceding the last row (step S601).

[0077] The game record generation unit 12 determines whether the number of moves is 1 or not (step S602). If the game record generation unit 12 determines in step S602 that the number of moves is 1 (step S602: Y), it determines that the moves are consecutive (step S603). On the other hand, if the game record generation unit 12 determines in step S602 that the number of moves is not 1 (it is a predetermined number of 2 or more) (step S602: N), it determines that the moves are discontinuous (step S604).

[0078] Returning to Figure 5, the game record generation unit 12, after moving from step S503, determines that the moves are not discontinuous (they are continuous) (step S504:N), generates the moves for the last row, outputs them to the game record embedding unit 13 (step S505), and then moves on to step S509.

[0079] Specifically, the game record generation unit 12 determines which player (first or second player) moved the piece, the type of piece moved, and how the piece moved (movement from square to square on the board, movement from the board to the piece stand, movement from the piece stand to square on the board) by calculating the difference between the board information and piece stand information corresponding to the move in the last row.

[0080] The game record generation unit 12 generates moves that reflect the players who moved the pieces, the type of piece moved, and how the piece moved, according to predetermined game record rules. The game record rules are regulations for representing information such as first player / second player, promotion / non-promotion, same, drop, approach / pull / up as moves, and the game record generation unit 12 generates, for example, "△37 pawn drop" as a move (see the location of the move in Figure 10 below when the number of moves is 60). This move "△37 pawn drop" indicates that the second player moved a piece of type "pawn" from their piece stand and dropped it on the square "37" on the board.

[0081] On the other hand, if the game record generation unit 12 determines that the moves are discontinuous (step S504:Y), it generates "The game record is skipped" as the move with move number n, where n is the number of moves in the last row, and outputs this to the game record embedding unit 13 (step S506). n is an integer of 2 or more.

[0082] The game record generation unit 12 generates n+1 moves based on the board information and piece stand information for the last row's move and the move immediately preceding the last row's move, which were extracted from the intermediate table T in step S502, and outputs this to the game record embedding unit 13 (step S507).

[0083] The game record generation unit 12 modifies the intermediate table T by copying the board information and piece stand information corresponding to the number of moves n in the last row of the intermediate table T to the new last row when the row with the number of moves n+1 is designated as the new last row, and deleting the information in the row with the number of moves n (step S508), storing the modified intermediate table T in the storage unit 20, and proceeding to step S509.

[0084] The game record generation unit 12 moves from step S505 or step S508 to add (write) the number of moves and the moves (the moves generated in steps S505, S506, and S507) to the game record data D1, and stores the updated game record data D1 in the storage unit 20 (step S509).

[0085] Figure 10 shows an example of game record data D1. This game record data D1 consists of multiple sets of data, where the number of moves and the moves themselves are considered as one set of data. Similar to Figure 9, the number of moves is a numerical value that starts from 0 and increments by 1, with the most recent move added to the bottom row.

[0086] Returning to Figure 5, the game record generation unit 12 reads the time from the clock 14 and, based on the start time of the game video A and the read time (the time when the move was generated), generates a time code for the game video A corresponding to the move generated in steps S505 and S507 (the elapsed time from the start time of the game video A to the time when the move was generated).

[0087] The game record generation unit 12 adds (writes) the number of moves, the moves (the moves generated in steps S505, S506, and S507) and the timecode to the timecode-equipped game record data D2, and stores the updated timecode-equipped game record data D2 in the storage unit 20 (step S510). If the move to which the number of moves, the moves, and the timecode are added to the timecode-equipped game record data D2 is the "game record is missing" generated in step S506, the corresponding timecode is set to null.

[0088] Figure 11 shows an example of timecoded game record data D2. This timecoded game record data D2 consists of multiple sets of data, where the number of moves, the move itself, and the timecode are considered as one set of data. Timecoded game record data D2 is the game record data D1 shown in Figure 10 with timecodes added for each move.

[0089] As a result, timecode-attached game record data D2, which links the move and the timecode at the time the move was made, is generated as metadata for the game video A.

[0090] <Game record embedding section 13> Next, we will explain the details of the game record embedding section 13 shown in Figure 1. Figure 7 is a flowchart showing an example of the processing of the game record embedding section 13.

[0091] The game record embedding unit 13 receives the game video A, the piece identification result from the board / piece identification unit 10, and the move from the game record generation unit 12 (step S701). Then, the game record embedding unit 13 generates piece information a, b, and c, which will be described later, based on the piece identification result (step S702).

[0092] As mentioned above, the piece identification result is information indicating the type and position of each piece held by the player playing first and the player playing second in the game video A.

[0093] The game record embedding unit 13 generates a game record embedded video B by embedding piece information a, b, c and moves in predetermined locations on the game video A at a predetermined frame rate (for example, 5 frames per second) (step S703). Then, the game record embedding unit 13 stores the game record embedded video B in the storage unit 20 (step S704).

[0094] Figure 12 shows an example of the screen of game video B with embedded game record. As shown in Figure 12, game video B with embedded game record is a video in which piece information a, b, c and moves are embedded in game video A.

[0095] Piece information a indicates the number of pieces identified by the board / piece identification unit 10 (recognized by the game record generation device 1) out of the total number of pieces (40), as well as the type and number of each piece. In the example in Figure 12, piece information a indicates that the number of pieces identified by the board / piece identification unit 10 is 40 (koma:40 / 40), the number of identified pieces of type "pawn" is 18 (fu:18), the number of identified pieces of type "lance" is 4 (kyousu:4), ..., and the number of pieces of type "king" is 1 (gyoku:1).

[0096] Piece information b shows the type and number of each piece held by the first and second players. In the example in Figure 12, piece information b shows that the second player holds 7 pawns (down_fu:n(7)), 2 silver pieces (down_gin:n(2)), ..., and the first player holds 2 lances (up_kyousu:n(2)).

[0097] Piece information c shows the type of piece and the likelihood (similarity) of the identified piece for each player, both the first and second players. In the example in Figure 12, the type and likelihood information is shown behind or in front of the piece. For example, the arrow indicates that the likelihood of the piece "Gold" owned by the first player and placed on that square on the board is 94% (up_kin 0.94). The likelihood of a piece is calculated by the board / piece identification unit 10.

[0098] In this way, the game record embedding unit 13 embeds information about the game at the time of the move (the move in question) into the game video A, and the game record embedded game video B is generated.

[0099] <Playback part 15> Next, we will explain the screen displayed by the playback unit 15 shown in Figure 1. Figure 13 is a diagram showing an example of a screen displayed by the playback unit 15.

[0100] Referring to Figure 13, the playback unit 15 displays on the screen the game video A, the number of moves, time codes (TC), and moves included in the timecode-equipped game record data D2, as well as the change in the AI ​​win rate for each move.

[0101] Furthermore, the game video A is played and paused according to user operations, and for example, when a move is specified, playback starts from the playback position corresponding to that move. In this case, when a move is specified by user operation, the playback unit 15 extracts the time code corresponding to that move from the timecode-attached game record data D2 and starts playback of the game video A from the playback position indicated by that time code.

[0102] Although not shown in Figure 13, the playback unit 15 displays the embedded game record video B on the screen. Similar to game record video A, embedded game record video B can be played and paused according to user operations.

[0103] This allows users to, for example, determine the time taken for each move by viewing timecoded game record data D2 of a shogi game video A according to their own instructions. They can also understand the changes in piece placement with each move and analyze the tactics of the players.

[0104] Furthermore, when a move is specified by the user, the timecode included in the timecode-equipped game record data D2 can be used to start playback of the game video A from the playback position corresponding to that move, thus making effective use of the timecode.

[0105] As described above, according to the game record generation device 1 of the embodiment of the present invention, the board and piece discrimination unit 10 uses the board learning model M1 to discriminate the board and piece stand from the game video A, and uses the piece learning model M2 to discriminate the pieces from the video of the board and piece stand, and generates a piece discrimination result.

[0106] The intermediate table generation unit 11 generates an intermediate table T consisting of the number of moves, board information, and piece stand information based on the piece identification result. The game record generation unit 12 generates the move for the latest number of moves based on the board information and piece stand information for the latest number of moves and the move immediately preceding the latest number of moves included in the intermediate table T, and generates game record data D1 consisting of the number of moves and the move. The game record generation unit 12 also generates a time code by reading the time when the move was generated from the clock 14, and generates timecoded game record data D2 consisting of the number of moves, the move, and the time code.

[0107] The game record embedding unit 13 generates piece information a, b, and c based on the piece identification result, and embeds the piece information a, b, c and moves into the game video A to generate the game video B with the embedded game record.

[0108] This allows for the generation of timecoded game record data D2 from the game video A, and the timecoded game record data D2 can be used to clearly indicate, for example, the time taken by each player to make a move. Therefore, there is no need to manually generate the timecoded game record data D2, which reduces labor during program recording and allows for the effective use of the game record data.

[0109] Although the present invention has been described above with reference to embodiments, the present invention is not limited to the above embodiments and can be modified in various ways without departing from the technical concept.

[0110] Furthermore, a standard computer can be used as the hardware configuration for the game record generation device 1 according to the embodiment of the present invention. The game record generation device 1 is composed of a computer equipped with a CPU, a volatile storage medium such as RAM, a non-volatile storage medium such as ROM, and an interface.

[0111] The functions of the game record generation device 1, including the board / piece discrimination unit 10, the intermediate table generation unit 11, the game record generation unit 12, the game record embedding unit 13, the clock 14, the playback unit 15, and the storage unit 20, are each realized by having the CPU execute a program that describes these functions.

[0112] These programs are stored in the aforementioned storage medium and are read and executed by the CPU. These programs can also be stored and distributed on storage media such as magnetic disks (HDDs, etc.), optical disks (CD-ROMs, DVDs, etc.), and semiconductor memory (SSDs, etc.), and can be transmitted and received over a network. [Explanation of Symbols]

[0113] 1 Game record generation device 10 Board and piece identification section 11 Intermediate Table Generation Unit 12 Game record generation department 13. Game record embedding section 14 Clocks 15 Playback Department 20 Memory section A Match Video M1 Panel-based learning model M2 learning model for chess pieces T Intermediate Table D1 Game Record Data D2 Timecoded Game Record Data B. Game record embedded in video Z threshold a,b,c piece information n, n+1 moves

Claims

1. In a game record generation device that generates game records based on video footage of shogi matches, A board and piece discrimination unit that uses a predetermined learning model for the board to discriminate the shogi board and piece stand from the video of the game, and uses a predetermined learning model for the pieces to discriminate the pieces from the video of the shogi board and the video of the piece stand, An intermediate table generation unit generates a set of data consisting of the number of moves, board information indicating the type and position of pieces on the shogi board, and piece stand information indicating the type and number of pieces on the piece stand, based on the type and position of the pieces included in the game video when the total number of pieces identified by the board and piece identification unit is 40, and writes the data of the set to an intermediate table, thereby generating the intermediate table consisting of multiple sets of data written in the order of the number of moves, A game record generation unit generates moves based on the intermediate table generated by the intermediate table generation unit, generates a time code for the elapsed time from the start time of the game video to the time the moves were generated using the internal clock of the game record generation device, and generates timecoded game record data consisting of the number of moves, the moves, and the time code. A game record generation device characterized by having the following features.

2. In the game record generation device according to claim 1, The aforementioned predetermined board learning model is trained to identify the shogi board and piece stand used in the game, as well as the shogi board and piece stand displayed in the wipe, from the game video, while preventing the identification of the shogi board used for the large-screen commentary. The aforementioned board and piece discrimination unit is, A game record generation device characterized by using the predetermined board learning model to determine from the game video the shogi board and piece stand used in the game, as well as the shogi board and piece stand displayed in the wipe.

3. In the game record generation device according to claim 1, The intermediate table generation unit, When the data for the aforementioned set is generated, the data for that set is written to the intermediate table as temporary data. Subsequently, when data for a new set is generated, if the data for the new set differs from the temporary data, the temporary data in the intermediate table is overwritten with the data for the new set as temporary data. A game record generation device characterized in that, if the provisional data included in the intermediate table remains unchanged for a predetermined time, the provisional data is confirmed as regular data to generate the intermediate table.

4. In the game record generation device according to claim 1, The aforementioned game record generation unit, A game record generation device characterized by generating moves based on the difference between the board information and piece stand information contained in the latest set of data included in the intermediate table and the board information and piece stand information contained in the set of data one step prior to the latest.

5. In the game record generation device according to claim 1, The latest data set included in the aforementioned intermediate table is designated as the first data, and the data set one step prior to the latest is designated as the second data. The aforementioned game record generation unit, Based on the first and second data, calculate the total number of pieces moved when the number of moves in the second data changes to the number of moves in the first data. If the sum is 1, it is determined that the change from the number of moves in the second data to the number of moves in the first data is continuous, the moves are generated, and the game record data with timecodes is generated. A game record generation device characterized by determining that the change is discontinuous when the sum of the above is 2 or more, generating a move indicating that the game record is skipped, and generating the game record data with timecode.

6. A program for causing a computer to function as a game record generation device according to any one of claims 1 to 5.

Images