Tactile presentation device and program

Abstract

To generate information for presenting haptic stimulus that contributes to improvement of immersiveness when a viewer views video.SOLUTION: A video analysis unit 11 of a haptic presentation device 1 is configured to: obtain a plurality of time-series frames by sampling video E; detect rectangular coordinates of one or more objects in each of the frames; calculate rectangular data v (a moving amount v1(n) of a center of a rectangle, a rectangle area v2(n), and a rectangle magnification v3(n)) of an object having a maximum area; extract maximum rectangle data vmax; and calculate a sound volume amplification amount a(n) on the basis of the rectangle data v and the maximum rectangle data vmax. A sound volume control unit 12 extracts a low-frequency sound signal S or the like from the video E, and multiples the sound volume amplification amount a(n) by sound volume A of the low-frequency sound signal S, to generate a new low-frequency sound signal S' with the amplified sound volume A. A haptic presentation unit 13 outputs the low-frequency sound signal S' to a haptic device 7.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a tactile presentation device and a program for generating information for presenting tactile stimulation to a viewer of a video via a tactile device. [Background technology]

[0002] While video has traditionally been a medium primarily related to the senses of sight and hearing, technology that provides tactile stimulation as a third sense has been attracting attention. For example, a body-sensing audio system is known, in which a chair is equipped with a mechanism that provides tactile stimulation linked to music, and a tactile device that converts sound into vibration is also known.

[0003] Specifically, this sensible acoustic system incorporates a vibrator into a chair, extracts low-frequency components from music, and converts the low-frequency components into tactile vibration information using the vibrator, thereby presenting a vibrational tactile stimulus to the listener (see, for example, Patent Document 1).

[0004] In addition to a haptic audio system in which a vibrator is built into a chair, there are also known technologies that present tactile stimuli such as vibrations and somatosensory stimuli such as a sense of movement in addition to conventional video and audio in theme parks, movie theaters, etc. There is also known a technology that transmits tactile information of recorded vibrations via communication in addition to the video and audio of television broadcasts by using a broadcasting and communication collaboration service.

[0005] Furthermore, in a game device that uses a visual display, a technology is known that provides the player with a bodily sensation along with the visual display (see, for example, Patent Document 2). Specifically, this game device outputs a signal amplified by a high-power amplifier to a low-frequency speaker at the timing of a specific visual display, and the low-frequency speaker provides the player with the bodily sensation of a low-frequency sound source.

[0006] Furthermore, a technology is known for a bodily sound system that does not cause discomfort or a feeling of oppression to the viewer even when used for a long period of time (see, for example, Patent Document 3). Specifically, this bodily sound system has a seat with a backrest and a seat portion, a band splitting circuit that splits an input audio signal into bands and outputs a first audio signal and a second audio signal, a first vibration element that vibrates in response to the first audio signal and is arranged in the backrest so that the vibration direction is parallel to the surface of the backrest facing the user, and a second vibration element that vibrates in response to the second audio signal and is arranged in the seat so that the vibration direction is parallel to the surface of the seat facing the user.

[0007] In this way, by providing stimulation to the third sense, tactile sensation, in addition to visual and auditory sensations while watching a video, it is possible to achieve a more immersive and realistic video viewing experience. In other words, by inputting an audio signal, converting it into tactile information, and presenting tactile stimulations continuously, it is possible to enhance the immersive and realistic sensation of the video content.

[0008] Attempts to convert such audio signals into tactile information and present tactile stimuli to viewers have been made for some time. Hereinafter, the method of inputting audio signals, converting them into tactile information, and presenting tactile stimuli will be referred to as the "audio input method."

[0009] An example of this "voice input method" is a chair-type tactile presentation system. This chair-type tactile presentation system is linked to an image of a vehicle such as a tram displayed at a 180-degree viewing angle on a flexible display, and a chair-type tactile device converts the audio signal into tactile information and presents tactile stimuli. This allows for a high level of immersion through visual stimuli from the image displayed on the flexible display and tactile stimuli from vibrations on the seat and feet. [Prior art documents] [Patent documents]

[0010] [Patent Document 1] Japanese Patent Application Publication No. 3-102499 [Patent Document 2] Japanese Patent Application Publication No. 6-339576 [Patent Document 3] Japanese Patent Application Laid-Open No. 2008-141477 Summary of the Invention [Problem to be solved by the invention]

[0011] The audio input method using the aforementioned bodily sound system converts audio signals into tactile information and presents tactile stimuli. Because this audio input method is based on actual audio, it is possible to present tactile stimuli that do not feel unnatural in combination with video and audio.

[0012] However, actual audio often includes background sounds such as environmental sounds and background music, so even if actual audio is used, it is not always possible to obtain appropriate vibrations.

[0013] Here, if it were possible to control vibrations in accordance with the movements of animals or other objects in the video, it would be possible to provide a powerful and immersive experience.

[0014] Therefore, the present invention has been made to solve the above-mentioned problems, and its purpose is to provide a tactile presentation device and program that generate information for presenting tactile stimuli that contribute to improving the sense of immersion when a viewer watches a video. [Means for solving the problem]

[0015] In order to solve the above problem, the tactile presentation device of claim 1 is a tactile presentation device that extracts low-frequency audio signals from video and generates information for presenting tactile stimuli to a viewer of the video via a tactile device based on the low-frequency audio signals, and is characterized by comprising: a video analysis unit that detects objects using a predetermined NN (neural network) for each of a plurality of time-series frames included in the video, calculates at least one piece of data from the object's movement amount, area, and magnification rate, and calculates a volume amplification amount based on the at least one piece of data; a volume control unit that extracts the low-frequency audio signals from the video and amplifies the volume of the low-frequency audio signals corresponding to the frames based on the volume amplification amount for the frames calculated by the video analysis unit; and a tactile presentation unit that outputs the low-frequency audio signals whose volume has been amplified by the volume control unit to the tactile device.

[0016] Furthermore, the tactile presentation device of claim 2 is characterized in that, in the tactile presentation device of claim 1, the video analysis unit detects the object using the specified NN for each of the plurality of frames, calculates the amount of movement, area and magnification rate of the object, and calculates the amount of volume amplification based on the amount of movement, area and magnification rate of the object.

[0017] The tactile presentation device of claim 3 is the tactile presentation device of claim 2, wherein the video analysis unit includes a frame sampling processing unit that samples the video into a plurality of frames in time series at predetermined intervals; an object detection processing unit that detects one or more objects using the predetermined NN for each of the plurality of frames sampled by the frame sampling processing unit and estimates rectangular coordinates including the object for each object; and a processing unit that selects an object with a largest area from the one or more objects based on the rectangular coordinates of each object estimated by the object detection processing unit for each of the plurality of frames, and generates first rectangular data regarding the amount of movement of the object and second rectangular data regarding the area of ​​the object based on the rectangular coordinates of the object in a predetermined number of frames including the frame in question. a rectangle calculation unit that calculates shape data and third rectangle data related to a magnification ratio of the object; a maximum value extraction unit that extracts a maximum value of the first rectangle data, a maximum value of the second rectangle data, and a maximum value of the third rectangle data from the first rectangle data, the second rectangle data, and the third rectangle data calculated by the rectangle calculation unit for the plurality of frames, respectively; and a volume amplification amount calculation unit that normalizes the first rectangle data, the second rectangle data, and the third rectangle data calculated by the rectangle calculation unit by the maximum value of the first rectangle data, the maximum value of the second rectangle data, and the maximum value of the third rectangle data extracted by the maximum value extraction unit, respectively, and calculates the volume amplification amount by weighting the normalized first rectangle data, the second rectangle data, and the third rectangle data.

[0018] The tactile presentation device of claim 4 is the tactile presentation device of claim 2, wherein the video analysis unit includes a frame sampling processing unit that samples the video into a plurality of frames in time series at predetermined intervals, an object detection processing unit that detects one or more objects for each of the plurality of frames sampled by the frame sampling processing unit using the predetermined NN and estimates rectangular coordinates including the object for each object, and an object detection processing unit that selects an object having a largest area from the one or more objects for each of the plurality of frames based on the rectangular coordinates of each object estimated by the object detection processing unit, and estimates a predetermined number of frames including the frame for the largest object. a rectangle calculation unit that calculates first rectangular data related to the amount of movement of the object, second rectangular data related to the area of ​​the object, and third rectangular data related to the magnification rate of the object based on rectangular coordinates of the object in the system; and a volume amplification amount calculation unit that normalizes the first rectangular data, the second rectangular data, and the third rectangular data calculated by the rectangle calculation unit by a predetermined maximum value of the first rectangular data, a predetermined maximum value of the second rectangular data, and a predetermined maximum value of the third rectangular data, respectively, and calculates the volume amplification amount by weighting the normalized first rectangular data, second rectangular data, and third rectangular data.

[0019] Furthermore, the tactile presentation device of claim 5 is a tactile presentation device according to any one of claims 1 to 4, characterized in that the volume control unit includes a low-frequency audio extraction unit that extracts the low-frequency audio signal, the video signal, and audio signals other than the low-frequency audio signal from the video; a volume amplification control unit that multiplies the volume amplification amount for the frame calculated by the video analysis unit by the low-frequency audio signal extracted by the low-frequency audio extraction unit corresponding to the frame to generate a new low-frequency audio signal; a synthesis unit that synthesizes the new low-frequency audio signal generated by the volume amplification control unit, the video signal extracted by the low-frequency audio extraction unit, and audio signals other than the low-frequency audio signal to obtain a volume-controlled video; and the tactile presentation unit extracts the new low-frequency audio signal from the volume-controlled video obtained by the synthesis unit and outputs the new low-frequency audio signal to the tactile device.

[0020] Furthermore, the program of claim 6 is characterized in that it causes a computer constituting a tactile presentation device that extracts low-frequency audio signals from video and generates information for presenting tactile stimuli to a viewer of the video via a tactile device based on the low-frequency audio signals to function as a video analysis unit that detects objects using a predetermined NN (neural network) for each of a plurality of time-series frames included in the video, calculates at least one piece of data from the object's movement amount, area, and magnification rate, and calculates a volume amplification amount based on the at least one piece of data, a volume control unit that extracts the low-frequency audio signals from the video and amplifies the volume of the low-frequency audio signals corresponding to the frames based on the volume amplification amount for the frames calculated by the video analysis unit, and a tactile presentation unit that outputs the low-frequency audio signals whose volume has been amplified by the volume control unit to the tactile device. [Effects of the Invention]

[0021] As described above, according to the present invention, it is possible to generate information for presenting tactile stimuli that contribute to improving the sense of immersion when a viewer is watching a video. [Brief explanation of the drawings]

[0022] [Figure 1] 1 is a block diagram illustrating an example of the configuration of a tactile presentation device according to an embodiment of the present invention. [Figure 2] 2 is a flowchart showing an example of processing performed by the tactile presentation device shown in FIG. [Figure 3] FIG. 2 is a block diagram showing an example of the configuration of a video analysis unit. [Figure 4] 4 is a flowchart showing an example of processing by the video analysis unit shown in FIG. 3. [Figure 5] 10 is a block diagram showing an example of the configuration of a volume amplification amount calculation processing unit. FIG. [Figure 6] FIG. 2 is a block diagram showing an example of the configuration of a volume control unit. [Figure 7] 7 is a flowchart showing an example of processing performed by the volume control unit shown in FIG. 6. [Figure 8] FIG. 10 is a diagram illustrating an example of the configuration of a haptic sense providing unit when playing back video E in 5.1ch format. [Figure 9] 10 is a diagram showing an example of a frame of video E and examples of rectangular coordinates k1, k2, and k3. [Figure 10] FIG. 10 is a diagram showing the calculation results of the volume amplification amount a(n). DETAILED DESCRIPTION OF THE INVENTION

[0023] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. [Tactile presentation device] FIG. 1 is a block diagram showing an example of the configuration of a tactile presentation device according to an embodiment of the present invention, and FIG. 2 is a flowchart showing an example of processing performed by the tactile presentation device shown in FIG.

[0024] This tactile presentation device 1 includes a video analysis unit 11, a volume control unit 12, and a tactile presentation unit 13. The tactile presentation device 1 calculates a volume amplification amount a proportional to the amount of movement of an object from the video E, controls the volume of the low-frequency audio signal S that is the source of vibration based on the volume amplification amount a to generate a volume-controlled video E', extracts the controlled low-frequency audio signal S' from the volume-controlled video E', and outputs it to the tactile device 7. This can improve the sense of immersion felt by the viewer when watching the video E.

[0025] The video analysis unit 11 receives video E including one or more objects (step S201). Then, the video analysis unit 11 samples the video E, detects one or more objects for each of a plurality of time-series frames included in the video E, selects the object with the largest area, and calculates rectangle data v relating to the amount of movement, area, and magnification of the object (step S202).

[0026] The plurality of time-series frames included in the video E may be all of the frames that make up the video E, or may be a group of frames sampled at predetermined intervals.

[0027] The video analysis unit 11 calculates a volume amplification amount a proportional to the amount of movement, etc., for each of the multiple frames based on the rectangle data v (step S203). Then, the video analysis unit 11 outputs the volume amplification amount a to the volume control unit 12.

[0028] As a result, rectangle data v is calculated and a volume amplification amount a is calculated for each of a plurality of time-series frames included in video E from the first frame to the last frame of video E. Details of video analysis unit 11 will be described later.

[0029] The volume control unit 12 receives the volume amplification amount a for each sampled frame from the video analysis unit 11 and stores the volume amplification amount a in the memory 41 (described later) (step S204). As a result, the volume amplification amount a for each of the multiple frames included in the video E is stored in the memory 41.

[0030] The volume control unit 12 determines whether or not an operation to start viewing of video E (the same video as video E input by the video analysis unit 11) has been performed in accordance with an operation by the viewer (step S205). If the volume control unit 12 determines in step S205 that an operation to start viewing has not been performed (step S205: N), the volume control unit 12 waits until the operation is performed.

[0031] If it is determined in step S205 that an operation to start viewing has been performed (step S205: Y), the volume control unit 12 inputs the video E (step S206). Then, the volume control unit 12 extracts the low-frequency audio signal S, the video signal, and audio signals other than the low-frequency audio signal S from the video E (step S207).

[0032] For example, if video E includes an audio signal of a low-frequency channel, the volume control unit 12 extracts the audio signal of that channel as a low-frequency audio signal S by extracting it from video E. Also, if video E is made up of a video signal and an audio signal, and the audio signal is a signal including high-frequency components and low-frequency components, the volume control unit 12 extracts the low-frequency components from the audio signal included in video E and extracts the low-frequency components as a low-frequency audio signal S.

[0033] The volume control unit 12 reads out, from the memory 41 described later, the volume amplification amount a for the frame of the video signal corresponding to the low-frequency audio signal S of the video E extracted in step S207 (step S208). As a result, the volume amplification amounts a stored in the memory 41 are read out in order corresponding to the low-frequency audio signal S.

[0034] The volume control unit 12 generates a new low-frequency audio signal S' by amplifying the volume A of the low-frequency audio signal S corresponding to the frame of the volume amplification amount a based on the volume amplification amount a read out from the memory 41 (step S209). Here, the low-frequency audio signal S corresponding to the volume amplification amount a is the audio signal from the frame of the volume amplification amount a to just before the next frame of that frame (the next frame of the multiple frames stored in the memory 41).

[0035] Here, the larger the value of the rectangle data v, the larger the value of the volume amplification amount a, and the smaller the value of the rectangle data v, the smaller the value of the volume amplification amount a. Because the rectangle data v is data related to the movement amount, area, and magnification rate of an object, the larger these data are, the larger the value of the volume amplification amount a, and the smaller these data are, the smaller the value of the volume amplification amount a.

[0036] The larger the value of the volume amplification amount a, the higher the volume A of the new low-frequency audio signal S' after amplification. new is louder than the volume A before amplification, and the smaller the value of the volume amplification amount a, the louder the volume A of the new low-frequency audio signal S' after amplification. new will be even quieter than the volume A before amplification.

[0037] In other words, the greater the amount of movement, area, and magnification of the object, the greater the volume A of the new amplified low-frequency audio signal S'. new In addition, the smaller the amount of movement, area, and magnification of the object, the smaller the volume A of the new low-frequency audio signal S' after amplification. new becomes smaller.

[0038] The volume control unit 12 obtains a volume-controlled image E' by combining the low-frequency audio signal S' generated in step S209 with the video signal extracted in step S207 and audio signals other than the low-frequency audio signal S (step S210). Then, the volume control unit 12 outputs the volume-controlled image E' to the haptic sense providing unit 13. The volume control unit 12 will be described in detail later.

[0039] The haptic sense providing unit 13 receives the volume-controlled video E' from the volume control unit 12, extracts a low-frequency audio signal S' from the volume-controlled video E', and outputs the low-frequency audio signal S' to the haptic device 7 (step S211). Details of the haptic sense providing unit 13 will be described later.

[0040] As a result, the greater the amount of movement, area, and magnification of the object, the greater the low-frequency audio signal S' with a higher volume A is input to the haptic device 7, and the greater the vibration can be. On the other hand, the smaller the amount of movement, area, and magnification of the object, the less the low-frequency audio signal S' with a lower volume A is input to the haptic device 7, and the greater the vibration can be. In other words, it is possible to control the vibration in accordance with the movement of an object (for example, an animal) in the video E, and provide the viewer with a powerful and immersive experience.

[0041] Therefore, when viewing the video E, the tactile presentation device 1 can generate information for presenting tactile stimuli that contribute to improving the sense of immersion, and the viewer can receive vibration stimuli in accordance with the movement of objects in the video E, thereby improving the sense of immersion. Note that in Figures 1 and 2, the volume amplification amount a is output from the video analysis unit 11 to the volume control unit 12, but in Figure 3 and other figures described below, it is assumed that the volume amplification amount a(n) is output. n indicates the frame number.

[0042] (Video Analysis Section 11) Next, a detailed description will be given of the video analysis unit 11 shown in Fig. 1. Fig. 3 is a block diagram showing an example of the configuration of the video analysis unit 11, and Fig. 4 is a flowchart showing an example of processing performed by the video analysis unit 11 shown in Fig. 3.

[0043] The video analysis unit 11 includes a frame sampling processing unit 21, an object detection processing unit 22, and a volume amplification amount calculation processing unit .

[0044] The frame sampling processing unit 21 receives an input of video E (step S401) and samples the video E into a plurality of frames in time series at predetermined intervals (step S402). By sampling at predetermined intervals, the load of calculations and other processes in the subsequent processes can be reduced. Here, the frame sampling processing unit 21 may sample all of the frames that make up the video E.

[0045] The frame sampling processing unit 21 outputs each of the multiple frames (frames 0, . . . , n, . . . , N) after sampling to the object detection processing unit 22. N is an integer equal to or greater than 1, and n satisfies 0≦n≦N. Frame n indicates the frame with frame number n.

[0046] The object detection processing unit 22 receives each of the multiple frames from the frame sampling processing unit 21. Then, for each of the multiple frames, the object detection processing unit 22 detects an object included in the frame using a predetermined NN (neural network) and estimates the rectangular coordinates and the identification class of each of the detected objects (step S403). Then, the object detection processing unit 22 outputs the rectangular coordinates of each object included in the frame to the volume amplification amount calculation processing unit 23.

[0047] The rectangle coordinates are used to identify a rectangle that contains the object to the maximum extent, and are composed of, for example, the coordinate values ​​of the upper left vertex and the lower right vertex. In this case, the rectangle coordinates may be composed of the coordinate value of the upper left vertex, the width in the x-axis direction (horizontal direction), and the height in the y-axis direction (vertical direction).

[0048] This gives the rectangular coordinates of each object for each of the N frames (rectangular coordinates of each object for frame 0, ..., rectangular coordinates of each object for frame n, ..., rectangular coordinates of each object for frame N).

[0049] The NN used by the object detection processing unit 22 is not limited to a specific network configuration. For example, a configuration similar to the object detection model shown in the following document or an improved version based on these architectures may be used. [Non-patent document] Ge Zheng, et al., “Yolox: Exceeding yolo series in 2021.”, arXiv preprint arXiv:2107.08430 (2021).

[0050] 9 is a diagram showing an example of a frame of video E and examples of rectangular coordinates k1, k2, and k3. For a frame of video E, the object detection processing unit 22 detects three objects included in the frame using a neural network, and estimates the rectangular coordinates k1, k2, and k3 including each object and the object's identification class (in this example, the identification class of all objects is "elephant").

[0051] 3 and 4, the volume amplification amount calculation processing unit 23 receives the rectangular coordinates of each object for each of the multiple frames, and then performs the processes of steps S404 to S409, which will be described later.

[0052] 5 is a block diagram showing an example of the configuration of the volume amplification amount calculation processing unit 23. The volume amplification amount calculation processing unit 23 includes an object selection unit 31, a rectangle calculation unit 32, a memory 33, a maximum value extraction unit 34, and a volume amplification amount calculation unit 35.

[0053] 4 and 5, object selection unit 31 receives rectangular coordinates of each object for each of a plurality of frames, calculates the rectangular area of ​​each object based on the rectangular coordinates of each object, and selects the object with the largest rectangular area from one or more objects included in the frame (step S404). Object selection unit 31 outputs the rectangular coordinates of the object with the largest rectangular area for each of a plurality of frames to rectangle calculation unit 32.

[0054] In the example shown in FIG. 9, the object selecting unit 31 selects the object having the rectangular coordinate k1 with the largest rectangular area from among three objects having rectangular coordinates k1, k2, and k3, respectively.

[0055] 4 and 5, the rectangle calculation unit 32 receives the rectangle coordinates of the object with the largest rectangle area for each of the multiple frames from the object selection unit 31. Then, for the object with the largest rectangle area for each of the multiple frames, the rectangle calculation unit 32 calculates rectangle data v(n) such as the amount of movement of the object based on the rectangle coordinates of the object (step S405).

[0056] Here, the rectangle data v(n) for frame n consists of the movement amount v1(n) of the rectangle center, the rectangle area v2(n), and the rectangle enlargement rate v3(n). The rectangle data v(n) is calculated from rectangle coordinates, which are used to identify the rectangle that best encompasses the object. Therefore, the movement amount v1(n) of the rectangle center, the rectangle area v2(n), and the rectangle enlargement rate v3(n) can be used to identify the movement amount, area, and enlargement rate of the object, respectively.

[0057] For example, the rectangle calculation unit 32 calculates the movement amount v1(n) of the rectangle center and the rectangle enlargement rate v3(n) in frame n using the rectangle coordinates of the object with the largest rectangle area in a predetermined number of frames before and after frame n (the same number of rectangle coordinates as the predetermined number). Also, the rectangle calculation unit 32 calculates the rectangle area v2(n) using the rectangle coordinates of the object with the largest rectangle area in frame n.

[0058] Here, if no object is detected by the object detection processing unit 22, the rectangle calculation unit 32 sets the rectangle data v(n)=0, that is, v1(n)=v2(n)=v3(n)=0.

[0059] In addition, the rectangle calculation unit 32 calculates the overlap rate between the object with the largest rectangular area in frame n-1 (referred to as object α) and the object with the largest rectangular area in frame n (referred to as object β) based on the coordinates of each rectangle.

[0060] When a frame n-1 containing an object α is superimposed on a frame n containing an object β, if the area of ​​the region where the objects α and β exist is S1 and the area of ​​the region where the objects α and β overlap is S2, the overlap rate is expressed by the following formula. [Number 1] Overlap rate = S2 / S1 (1)

[0061] If the overlap rate is equal to or less than a predetermined value (e.g., 0.75), the rectangle calculation unit 32 determines that the objects α and β are different, and sets the movement amount v1(n) of the rectangle center of object β in frame n to 0, and also sets the rectangle enlargement rate v3(n) to 0. In other words, if the object with the largest rectangle area changes (if it changes to a different object of the same identification class, or if it changes to an object of a different identification class), the movement amount v1(n) of the rectangle center and the rectangle enlargement rate v3(n) are reset. On the other hand, if the overlap rate is greater than a predetermined value (e.g., 0.75), the rectangle calculation unit 32 determines that the objects α and β are the same.

[0062] The rectangle calculation unit 32 stores the rectangle data v(n), that is, the shift amount v1(n) of the rectangle center, the rectangle area v2(n), and the rectangle enlargement rate v3(n), for each of the multiple frames in the memory 33 (step S406).

[0063] As a result, memory 33 stores rectangular data v(n) of multiple frames (all frames) in time series sampled from video E. That is, memory 33 stores rectangular data v(0) of frame 0 (v1(0), v2(0), v3(0)), ..., rectangular data v(n) of frame n (v1(n), v2(n), v3(n)), ..., rectangular data v(N) of frame N (v1(N), v2(N), v3(N)).

[0064] When the rectangular data v(n) of all sampled frames is stored in the memory 33, the maximum value extraction unit 34 reads out the rectangular data v(n) of all sampled frames from the memory 33. Then, the maximum value extraction unit 34 extracts the maximum values ​​for each of the rectangle center movement amount v1(n), rectangle area v2(n), and rectangle enlargement rate v3(n) included in the rectangular data v(n), and outputs the maximum rectangular data v max is set (step S407).

[0065] Specifically, the maximum value extraction unit 34 extracts the maximum movement amount of the rectangle center from among the movement amounts v1(0), . . . , v1(n), . . . , v1(N) of the rectangle centers in all frames, and designates this as v 1max The maximum value extraction unit 34 extracts the maximum rectangular area from among the rectangular areas v2(0), . . . , v2(n), . . . , v2(N) in all frames, and designates this as v 2max The maximum value extraction unit 34 extracts the maximum rectangular enlargement ratio among the rectangular enlargement ratios v3(0), . . . , v3(n), . . . , v3(N) in all frames, and designates this as v 3max Then, the maximum value extraction unit 34 extracts the maximum movement amount v of the rectangle center. 1max , maximum rectangular area v 2max and the maximum rectangular expansion rate v 3max The maximum rectangular data v consisting of max Set.

[0066] The maximum value extraction unit 34 extracts the maximum rectangular data v max The maximum displacement of the rectangle center is v 1max , maximum rectangular area v 2max and the maximum rectangular expansion rate v 3max is output to the volume amplification amount calculation unit 35.

[0067] The volume amplification amount calculation unit 35 extracts the maximum rectangular data v from the maximum value extraction unit 34. max The volume amplification amount calculation unit 35 also reads out the rectangular data v(n) from the memory 33 for all sampled frames in order from the beginning, and calculates the rectangular data v(n) as the maximum rectangular data v max and calculates the volume amplification amount a(n) based on the normalized rectangular data v(n) (step S408). Then, the volume amplification amount calculation unit 35 outputs the volume amplification amount a(n) to the volume control unit 12 (step S409).

[0068] Specifically, the volume amplification amount calculation unit 35 calculates the amount of movement v1(n) of the rectangle center read from the memory 33 by multiplying the maximum amount of movement v 1maxThe volume amplification amount calculation unit 35 normalizes the rectangular area v2(n) read from the memory 33 by the maximum rectangular area v 2max The volume amplification calculation unit 35 normalizes the rectangle expansion ratio v3(n) read from the memory 33 by the maximum rectangle expansion ratio v 3max and calculate the normalized rectangular expansion ratio v'3(n).

[0069] The volume amplification amount calculation unit 35 calculates the volume amplification amount a(n) of frame n using the following formula. [Number 2] a(n)=(w1v'1(n)+w2v'2(n)+w3v'3(n)) / (w1+w2+w3) ···(2) w1, w2, and w3 are weighting coefficients for the preset shift amount v1(n) of the rectangle center, the rectangle area v2(n), and the rectangle enlargement rate v3(n), respectively.

[0070] As a result, the video analysis unit 11 obtains the volume amplification amount a(n) for each of a plurality of time-series frames sampled from the video E at predetermined intervals.

[0071] 10 is a diagram showing the calculation results of the volume amplification amount a(n), and shows the volume amplification amount a(n) calculated by the video analysis unit 11. The vertical axis represents the volume amplification amount a(n), and the horizontal axis represents time (frame number: n).

[0072] The video analysis unit 11 calculates the volume amplification amount a(n) shown in FIG. 10, and outputs the volume amplification amount a(n) to the volume control unit 12 at the subsequent stage.

[0073] (Volume control unit 12) Next, a detailed description will be given of the volume control unit 12 shown in Fig. 1. Fig. 6 is a block diagram showing an example of the configuration of the volume control unit 12, and Fig. 7 is a flowchart showing an example of the processing of the volume control unit 12 shown in Fig. 6.

[0074] The volume control unit 12 includes a memory 41, a low-frequency audio extraction unit 42, a volume amplification control unit 43, and a synthesis unit 44.

[0075] The volume control unit 12 receives the volume amplification amount a(n) for each sampled frame from the video analysis unit 11 and stores the volume amplification amount a(n) in the memory 41 (step S701). As a result, the volume amplification amounts a(n) for multiple frames (all frames) in time series sampled from the video E are stored in the memory 41.

[0076] The volume control unit 12 determines whether or not an operation to start viewing of video E (the same video as video E input by the video analysis unit 11 shown in FIG. 1) has been performed in accordance with an operation by the viewer (step S702). If the volume control unit 12 determines in step S702 that an operation to start viewing has not been performed (step S702: N), the volume control unit 12 waits until the operation is performed.

[0077] If the volume control unit 12 determines in step S702 that an operation to start viewing has been performed (step S702: Y), the low-frequency audio extraction unit 42 inputs the video E and extracts the low-frequency audio signal S, the video signal, and audio signals other than the low-frequency audio signal S from the video E (step S703).

[0078] In this case, the low-frequency audio extraction unit 42 may extract, as the low-frequency audio signal S, an audio signal from a sound source that has recorded only low frequencies in advance.

[0079] For example, in video E consisting of an 8K video signal and a 22.2ch audio signal, a low-frequency audio of 120Hz or less called an LFE (Low Frequency Effect) channel is used in 0.2ch. In this case, the low-frequency audio extraction unit 42 extracts this LFE audio signal as the low-frequency audio signal S, and by using the LFE audio signal in the downstream haptic device 7, the viewer can receive a tactile stimulus that is appropriate for the vibration.

[0080] Furthermore, if low-frequency audio such as LFE is not available, a mixed audio signal may be used. That is, the low-frequency audio extraction unit 42 can generate a pseudo low-frequency audio signal S by equalizing the mixed audio signal so as to emphasize the low-frequency components and suppress the high-frequency components.

[0081] The low-frequency audio extraction unit outputs the low-frequency audio signal S to the volume amplification control unit 43, and outputs the video signal and audio signals other than the low-frequency audio signal S (other audio signals) to the synthesis unit .

[0082] The volume amplification control unit 43 receives the low-frequency audio signal S from the low-frequency audio extraction unit 42 and reads out the volume amplification amounts a(n) for all sampled frames from the memory 41 in order from the first (step S704). Then, the volume amplification control unit 43 amplifies the volume A of the low-frequency audio signal S based on the volume amplification amount a(n) read out from the memory 41 to generate a new low-frequency audio signal S' (step S705). The volume amplification control unit 43 outputs the new low-frequency audio signal S' to the synthesis unit 44.

[0083] For example, the volume amplification control unit 43 amplifies the volume A of the low-frequency audio signal S of the frame corresponding to the volume amplification amount a(n) so that the volume A is proportional to the volume amplification amount a(n) according to the following formula: new A new low frequency audio signal S' having the following characteristics is generated. [Number 3] A new = a(n)A + c (3)

[0084] where c is the volume A new is a constant (minimum value) for raising the level of the low-frequency audio signal S of the frame corresponding to the volume amplification amount a(n) is the audio signal from the time point of the frame corresponding to the volume amplification amount a(n) to the time point immediately before the next frame.

[0085] As a result, the smaller the volume amplification amount a(n), i.e., the smaller the movement amount v1(n) of the rectangular center of the object with the largest rectangular area, the rectangular area v2(n), and the rectangular expansion rate v3(n), the smaller the volume A of the low-frequency audio signal S. On the other hand, the larger the volume amplification amount a(n), i.e., the larger the movement amount v1(n) of the rectangular center of the object with the largest rectangular area, the rectangular area v2(n), and the rectangular expansion rate v3(n), the larger the volume A of the low-frequency audio signal S.

[0086] The synthesis unit 44 receives the low-frequency audio signal S' from the volume amplification control unit 43, and also receives the video signal and audio signals other than the low-frequency audio signal S from the low-frequency audio extraction unit 42. The synthesis unit 44 then synthesizes the low-frequency audio signal S', the video signal, and the audio signals other than the low-frequency audio signal S to obtain a volume-controlled image E' (step S706). The synthesis unit 44 outputs the volume-controlled image E' to the haptic presentation unit 13 (step S707).

[0087] As a result, when the volume amplification amount a(n) in the frame of the video E is small, the volume control unit 12 reduces the volume A of the low-frequency audio signal S, thereby obtaining a new volume A new In addition, when the volume amplification amount a(n) in the frame of the video E is large, the volume A of the low-frequency audio signal S is increased to generate a volume-controlled video E' including the low-frequency audio signal S'. new A volume-controlled image E' containing the low-frequency audio signal S' is generated.

[0088] 7 may further include a smoothing unit before the memory 41. The smoothing unit receives the volume amplification amount a(n) for each sampled frame from the video analysis unit 11.

[0089] If the volume amplification amount a(n) is unstable (for example, if the rate of change of the volume amplification amount a(n) is equal to or greater than a predetermined value), the smoothing unit smoothes the volume amplification amount a(n) of the frame using a predetermined number of frames before and after (nearby).The smoothing unit then stores the smoothed volume amplification amount a(n) in memory 41.

[0090] (Tactile presentation unit 13) Next, the tactile sense providing unit 13 shown in Fig. 1 will be described in detail. Fig. 8 is a diagram showing an example of the configuration of the tactile sense providing unit 13 when playing back video E in 5.1ch format. This example shows the case where audio signals of three channels, L, R, and LFE, of audio signals in 5.1ch format (L, R, C, SL, SR, LFE) are used.

[0091] The tactile sense providing unit 13 includes an extracting unit 51 and an amplifying unit 52. Note that in Fig. 8, the configuration for amplifying the video signal and the L and R audio signals is omitted.

[0092] The extraction unit 51 inputs the volume-controlled video E' from the volume control unit 12, and extracts an LFE audio signal as a low-frequency audio signal S' from the volume-controlled video E', as well as a video signal and L (left) and R (right) audio signals. The extraction unit 51 outputs the LFE audio signal to the amplification unit 52, and the amplification unit 52 amplifies the LFE audio signal and outputs it to the haptic device 7 and the speaker 9. The extraction unit 51 also outputs the video signal to the display 8, and outputs the L and R audio signals to the speaker 9.

[0093] The haptic device 7 receives an LFE audio signal from the amplifier 52, and presents a smaller vibration to the viewer as the volume of the LFE audio signal decreases, and presents a larger vibration to the viewer as the volume of the LFE audio signal increases.

[0094] This allows the viewer to watch video E and receive tactile stimulation linked to video E through tactile device 7, which inputs LFE audio signals from tactile presentation unit 13, display 8, which inputs video signals, and speaker 9, which inputs L, R, and LFE audio signals.

[0095] In particular, in a frame of video E, when the movement amount v1(n) of the rectangular center of the object with the largest rectangular area, the rectangular area v2(n), and the rectangular expansion rate v3(n) are small, the volume A of the low-frequency audio signal S is reduced, allowing the viewer to receive a weaker tactile stimulus than usual. On the other hand, when the movement amount v1(n), the rectangular area v2(n), and the rectangular expansion rate v3(n) of the object with the largest rectangular area are large, the volume A of the low-frequency audio signal S is increased, allowing the viewer to receive a stronger tactile stimulus than usual.

[0096] Here, the low-frequency audio signal S', which is an LFE audio signal, is output to the tactile device 7 and converted into a tactile stimulus because, generally, humans can only receive tactile stimuli at low frequencies of around 200 Hz or less, and cannot receive appropriate tactile stimuli when the frequency of the audio signal is high.

[0097] In addition, the extraction unit 51 may extract an LFE audio signal as a low-frequency audio signal S' from the volume-controlled video E', as well as extract the video signal and L (left) and R (right) audio signals, output the LFE audio signal to the haptic device 7 via the amplification unit 52, output the video signal to the display 8, and output the L (left) and R (right) audio signals to the speaker 9.

[0098] As described above, according to the tactile presentation device 1 of the embodiment of the present invention, the video analysis unit 11 samples the video E to acquire multiple frames in time series, detects the rectangular coordinates of one or multiple objects for each of the multiple frames, selects the object with the largest area, and calculates the rectangular data v of the object (the movement amount v1(n) of the rectangle center, the rectangular area v2(n), and the rectangular enlargement rate v3(n)). Then, the video analysis unit 11 selects the maximum rectangular data v from the rectangular data v of all the sampled frames. max Extract the rectangular data v and the maximum rectangular data v max The volume amplification amount a(n) is calculated based on this.

[0099] When an operation to start viewing is performed, the volume control unit 12 extracts the low-frequency audio signal S etc. from the video E, and multiplies the volume A of the low-frequency audio signal S by the volume amplification amount a(n) to generate a new low-frequency audio signal S' by amplifying the volume A of the low-frequency audio signal S. Then, the volume control unit 12 synthesizes a volume-controlled video E' including the low-frequency audio signal S'.

[0100] The haptic sense providing unit 13 extracts a low-frequency audio signal S′ from the volume-controlled video E′ and outputs the low-frequency audio signal S′ to the haptic device 7.

[0101] As a result, the greater the movement amount v1(n) of the rectangle center of the object included in the frame, the greater the rectangle area v2(n) and the rectangle expansion rate v3(n), the greater the volume amplification amount a(n). new A low-frequency audio signal S' of the above value is generated, and the vibration caused by the haptic device 7 can be increased. On the other hand, the smaller the movement amount v1(n) of the rectangular center of the object included in the frame, the smaller the rectangular area v2(n) and the rectangular expansion rate v3(n), the smaller the volume amplification amount a(n) becomes. new A low frequency audio signal S' is generated, and the vibration caused by the haptic device 7 can be reduced.

[0102] That is, the viewer can receive vibration stimulation through the haptic device 7 in accordance with the movement of an object included in the video E. For example, the greater the movement of the object, the greater the vibration stimulation the viewer can receive, and the smaller the movement of the object, the smaller the vibration stimulation the viewer can receive.

[0103] Therefore, when viewing video E, the tactile presentation device 1 can generate information for presenting tactile stimuli that contribute to improving the sense of immersion, and the viewer can receive vibration stimuli in accordance with the movement of objects in video E, thereby improving the sense of immersion.

[0104] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above-described embodiments and can be modified in various ways without departing from the technical concept thereof.

[0105] For example, in the embodiment, the volume amplification amount calculation unit 35 of the volume amplification amount calculation processing unit 23 provided in the video analysis unit 11 normalizes the movement amount v1(n) of the object's rectangular center, the rectangular area v2(n), and the rectangular enlargement rate v3(n) by their respective maximum values, and calculates the volume amplification amount a(n) by weighting the normalized movement amount v'1(n), the rectangular area v'2(n), and the rectangular enlargement rate v'3(n). Alternatively, the volume amplification amount calculation unit 35 may calculate the volume amplification amount a(n) using data on at least one of the movement amount v1(n) of the rectangle center, the rectangular area v2(n), and the rectangular enlargement rate v3(n).

[0106] For example, the volume amplification amount calculation unit 35 normalizes two pieces of data from among the movement amount v1(n) of the rectangle center, the rectangle area v2(n), and the rectangle expansion rate v3(n) by their respective maximum values, and calculates the volume amplification amount a(n) by weighting the two normalized pieces of data.

[0107] Furthermore, the volume amplification amount calculation unit 35 normalizes one of the data of the rectangle center movement amount v1(n), the rectangle area v2(n), and the rectangle enlargement rate v3(n) by the maximum value thereof, and sets the normalized data as the volume amplification amount a(n). In this case, the volume amplification amount calculation unit 35 may calculate the volume amplification amount a(n) so that it is proportional to the value of the normalized data.

[0108] 2 and 7, the rectangle calculation unit 32 of the volume amplification amount calculation processing unit 23 provided in the video analysis unit 11 of the tactile presentation device 1 stores the rectangle data v(n) of all sampled frames in the memory 33. Then, the maximum value extraction unit 34 reads the rectangle data v(n) of all sampled frames from the memory 33 and extracts the maximum rectangle data v max The volume amplification calculation unit 35 extracts the rectangular data v(n) and the maximum rectangular data v maxThen, when the viewer starts viewing, the volume amplification control unit 43 of the volume control unit 12 amplifies the volume A of the low-frequency audio signal S based on the volume amplification amount a(n).

[0109] In response to this, the volume amplification amount calculation unit 35 calculates the maximum rectangular data v extracted by the maximum value extraction unit 34. max Instead of using the maximum rectangular data v max In this case, the volume amplification amount calculation processing unit 23 of the video analysis unit 11 does not need to include the memory 33 and the maximum value extraction unit 34 in the configuration example shown in FIG.

[0110] In other words, the volume control unit 12 does not wait for the viewer to start viewing, and the volume amplification control unit 43 amplifies the volume A of the low-frequency audio signal S based on the volume amplification amount a(n) calculated by the time interleaving processing unit 35.

[0111] This allows the entire process to be performed in real time, from the video analysis unit 11 inputting the video E and calculating the volume amplification amount a(n), to the volume control unit 12 amplifying the volume A of the low-frequency audio signal S to generate the volume-controlled video E', to the tactile presentation unit 13 outputting the low-frequency audio signal S' to the tactile device 7.

[0112] In addition, in the examples shown in Figures 2 and 7, when the volume control unit 12 of the tactile presentation device 1 determines that the viewer has performed an operation to start viewing, it extracts a low-frequency audio signal S, etc. from the video E and amplifies the volume A of the low-frequency audio signal S to generate a volume-controlled video E', and the tactile presentation unit 13 extracts the low-frequency audio signal S' from the volume-controlled video E' and outputs it to the tactile device 7.

[0113] In response to this, the volume control unit 12 may store the generated volume-controlled image E' in a memory not shown in Figure 6, and the tactile presentation unit 13 may repeatedly use the volume-controlled image E' stored in the memory each time the viewer starts viewing.

[0114] A normal computer can be used as the hardware configuration of the tactile presentation device 1. The tactile presentation device 1 is configured by a computer equipped with a CPU, a volatile storage medium such as RAM, a non-volatile storage medium such as ROM, an interface, etc.

[0115] The functions of the video analysis unit 11, the volume control unit 12, and the tactile presentation unit 13 provided in the tactile presentation device 1 are each realized by causing a CPU to execute a program in which these functions are written.

[0116] These programs are stored in the storage medium and are read and executed by the CPU. These programs can also be stored in a storage medium such as a magnetic disk (e.g., a floppy disk, a hard disk), an optical disk (e.g., a CD-ROM, a DVD), or a semiconductor memory and distributed, or can be transmitted and received via a network. [Explanation of symbols]

[0117] 1. Tactile presentation device 7. Haptic Devices 8. Display 9 Speakers 11 Video Analysis Section 12 Volume control 13 Tactile display unit 21 Frame sampling processing section 22 Object detection processing unit 23 Volume amplification amount calculation processing unit 31 Object selection section 32 Rectangle calculation part 33,41 memory 34 Maximum value extraction section 35 Volume amplification calculation section 42 Low-frequency audio extraction unit 43 Volume amplification control section 44 Synthesis section 51 Extraction part 52 Amplification section E-Video E' Volume controlled video k1,k2,k3 Rectangular coordinates a Volume amplification amount v Rectangular data v max Maximum rectangular data v1(n) Movement amount of the center of the rectangle v2(n) rectangular area v3(n) Rectangle expansion rate S,S' low frequency audio signal A,A new volume α,β Object

Claims

1. 1. A tactile presentation device that extracts a low-frequency audio signal from a video and generates information for presenting a tactile stimulus to a viewer of the video via a tactile device based on the low-frequency audio signal, a video analysis unit that detects an object using a predetermined neural network (NN) for each of a plurality of time-series frames included in the video, calculates at least one data item among a movement amount, an area, and an enlargement rate of the object, and calculates a volume amplification amount based on the at least one data item; a volume control unit that extracts the low-frequency audio signal from the video and amplifies the volume of the low-frequency audio signal corresponding to the frame based on the volume amplification amount for the frame calculated by the video analysis unit; a haptic presentation unit that outputs the low-frequency audio signal, the volume of which has been amplified by the volume control unit, to the haptic device; A tactile presentation device comprising:

2. The tactile presentation device according to claim 1 , The video analysis unit A tactile presentation device characterized in that, for each of the plurality of frames, the object is detected using the specified NN, the amount of movement, area and magnification rate of the object are calculated, and the amount of volume amplification is calculated based on the amount of movement, area and magnification rate of the object.

3. The tactile presentation device according to claim 2, The video analysis unit a frame sampling processing unit that samples the video into a plurality of frames in time series at predetermined intervals; an object detection processing unit that detects one or more objects using the predetermined neural network for each of the plurality of frames sampled by the frame sampling processing unit, and estimates rectangular coordinates including each object; a rectangle calculation unit that selects, for each of the plurality of frames, an object having a largest area from among the one or more objects based on the rectangular coordinates of each of the objects estimated by the object detection processing unit, and calculates, for the largest object, first rectangular data relating to an amount of movement of the object, second rectangular data relating to an area of ​​the object, and third rectangular data relating to an enlargement rate of the object based on the rectangular coordinates of the object in a predetermined number of frames including the frame; a maximum value extraction unit that extracts a maximum value of the first rectangle data, a maximum value of the second rectangle data, and a maximum value of the third rectangle data from the first rectangle data, the second rectangle data, and the third rectangle data calculated by the rectangle calculation unit in the plurality of frames, respectively; a volume amplification amount calculation unit that normalizes the first rectangle data, the second rectangle data, and the third rectangle data calculated by the rectangle calculation unit by the maximum value of the first rectangle data, the maximum value of the second rectangle data, and the maximum value of the third rectangle data extracted by the maximum value extraction unit, respectively, and calculates the volume amplification amount by weighting the normalized first rectangle data, the second rectangle data, and the third rectangle data; A tactile presentation device comprising:

4. The tactile presentation device according to claim 2, The video analysis unit a frame sampling processing unit that samples the video into a plurality of frames in time series at predetermined intervals; an object detection processing unit that detects one or more objects using the predetermined neural network for each of the plurality of frames sampled by the frame sampling processing unit, and estimates rectangular coordinates including each object; a rectangle calculation unit that selects, for each of the plurality of frames, an object having a largest area from among the one or more objects based on the rectangular coordinates of each of the objects estimated by the object detection processing unit, and calculates, for the largest object, first rectangular data relating to an amount of movement of the object, second rectangular data relating to an area of ​​the object, and third rectangular data relating to an enlargement rate of the object based on the rectangular coordinates of the object in a predetermined number of frames including the frame; a volume amplification amount calculation unit that calculates the volume amplification amount by normalizing the first rectangle data, the second rectangle data, and the third rectangle data calculated by the rectangle calculation unit by a predetermined maximum value of the first rectangle data, a predetermined maximum value of the second rectangle data, and a predetermined maximum value of the third rectangle data, respectively, and weighting the normalized first rectangle data, second rectangle data, and third rectangle data; A tactile presentation device comprising:

5. The tactile presentation device according to any one of claims 1 to 4, The volume control unit a low-frequency audio extraction unit that extracts the low-frequency audio signal, the video signal, and an audio signal other than the low-frequency audio signal from the video; a volume amplification control unit that multiplies the volume amplification amount for the frame calculated by the video analysis unit by the low-frequency audio signal extracted by the low-frequency audio extraction unit corresponding to the frame to generate a new low-frequency audio signal; a synthesis unit that synthesizes the new low-frequency audio signal generated by the volume amplification control unit, the video signal extracted by the low-frequency audio extraction unit, and an audio signal other than the low-frequency audio signal, to obtain a volume-controlled video; The tactile sense providing unit is A tactile presentation device characterized in that the new low-frequency audio signal is extracted from the volume-controlled video obtained by the synthesis unit, and the new low-frequency audio signal is output to the tactile device.

6. a computer constituting a tactile presentation device that extracts a low-frequency audio signal from a video and generates information for presenting a tactile stimulus to a viewer of the video via a tactile device based on the low-frequency audio signal; a video analysis unit that detects an object using a predetermined neural network (NN) for each of a plurality of time-series frames included in the video, calculates at least one piece of data among a movement amount, an area, and an enlargement rate of the object, and calculates a volume amplification amount based on the at least one piece of data; a volume control unit that extracts the low-frequency audio signal from the video and amplifies the volume of the low-frequency audio signal corresponding to the frame based on the volume amplification amount for the frame calculated by the video analysis unit; and a program for causing the haptic presentation unit to function as a haptic presentation unit that outputs the low-frequency audio signal, the volume of which has been amplified by the volume control unit, to the haptic device;

Images