Acoustic testing method and apparatus

Abstract

To enable ex-post verification to be conducted when determined to be an abnormal sound in the inspection of an abnormal sound such as a horn sound.SOLUTION: An inspection device comprises a sound acquisition unit that includes a microphone, and a video acquisition unit that is composed of a video camera that acquires a video overviewed from the front of a vehicle. When determined to be a horn sound, sound data in length of about several seconds is cut out and subjected to frequency analysis to determine whether a normal sound or an abnormal sound. When found to be an abnormal sound, video data in length of appropriately the same several seconds is cut out and preserved together the sound data. Sound data 70b that includes a time indicator 70b1, a spectrum 70c, a determination result table 70g, an overall vehicle image 70e, and an enlarged driver's seat image 70f for confirming horn switch operation are collected shown to a display. The sound is reproduced in response to a reproduction request, and the enlarged image 70f is reproduced in synchronism therewith.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] This invention relates to a method and an apparatus for acoustic inspection that performs inspection based on sound, such as in an inspection process of an automobile.

Background Art

[0002] For example, in the final inspection process of a completed vehicle on an automobile production line, an inspector performs a test drive of the completed vehicle to be inspected on a free roller, and conducts inspections on a number of items such as the engine, meters, brakes, horn, lights, etc. Generally, in this inspection process, regarding various sounds emitted from each part of the vehicle, the presence or absence of abnormalities is determined by a so-called sensory evaluation based on the inspector's sense. For example, for the horn, an inspection is performed where the inspector sounds the horn and listens to the sound himself / herself to confirm that it is a normal sound. Similarly, the sounds generated from the wiper blade during the operation of the wiper and the sounds generated from the brakes during brake operation are also subject to sensory inspection.

[0003] Instead of such sensory inspection, attempts have been made conventionally to acquire sound with a microphone and detect abnormalities by analyzing the signal.

[0004] Patent Document 1 discloses an abnormal sound diagnosis system that, in a belt-shaped production line where inspection objects such as automobile engines are conveyed side by side, acquires the sound generated by test-operating the movable parts of the inspection object as a measurement sound, and compares this measurement sound with a reference sound to determine whether it is a normal sound. An inspector located in front of the production line wears a head-mounted display equipped with a microphone, and can view the determination result of whether it is a normal sound in the head-mounted display, and can visually recognize the measurement range in a superimposed form on the inspection object. When it is displayed in the head-mounted display that it is an abnormal sound, the inspector attaches a label indicating a defective product to the inspection object.

Prior Art Documents

Patent Documents

[0005] [Patent Document 1] Japanese Patent Publication No. 2010-197361 [Overview of the project] [Problems that the invention aims to solve]

[0006] In many types of acoustic inspections, such as horn inspections during the final vehicle inspection process, so-called "traceability" is required, which allows for retrospective verification of inspections that have been judged to be abnormal. For example, in horn sound inspections, it is necessary to verify whether the inspector sounded the horn correctly. It is possible that the sound of a horn from another adjacent inspection line may be mixed in at a different timing than the inspector's horn operation, leading to a misjudgment of an abnormal sound.

[0007] The technology described in Patent Document 1 does not consider any post-test verification of such inspection results. [Means for solving the problem]

[0008] The acoustic inspection method according to this invention is: The system acquires sound emitted from the equipment being tested and generates sound data that includes the target sound. Simultaneously with sound acquisition, video of the equipment under inspection is acquired, and video data including the operation or action that caused the sound to be generated is generated. The sound data is analyzed to determine whether the target sound is an abnormal sound. If the system detects at least an abnormal sound, it will respond to the playback request by playing the audio data through the speaker, and simultaneously display the time-series audio data, including an indicator synchronized with the playback, the analysis results of the audio data, and the playback image of the video data synchronized with the audio data, all on a single screen.

[0009] For example, while sound emitted from the vehicle or other equipment being inspected is acquired via a microphone, video footage of the vehicle or other equipment being inspected is acquired via one or more cameras. Sound data, including, for example, a horn sound, is generated, and by analyzing this data, a determination is made as to whether the sound is abnormal. Meanwhile, if, for example, it is a horn sound, video data is generated that allows us to visually confirm the inspector's action that caused the horn sound (pressing the horn switch on the steering wheel).

[0010] For example, if playback is requested for post-event verification, audio data including the target sound (e.g., a horn sound) is played back via the speaker. Simultaneously, the screen displays a time-series audio data display with an indicator synchronized with the playback, the analysis results of the audio data, and a playback image of video data synchronized with the audio data playback. For example, in the case of a horn sound, video data showing the operation of pressing the horn switch is displayed in sync with the playback of the audio data. The audio data is also visually displayed as a time-series sound pressure waveform, etc., and this display of audio data includes an indicator synchronized with the playback, allowing users to visually identify where in the time-series audio data the sound currently being heard corresponds. [Effects of the Invention]

[0011] According to this invention, in response to a playback request, sound data is played back via a speaker, and a time-series display of the sound data including an indicator synchronized with the playback, the analysis results of the sound data, and the playback image of video data synchronized with the playback of the sound data are all displayed together, making it easy to verify the results of the sound inspection afterward. [Brief explanation of the drawing]

[0012] [Figure 1] A functional block diagram of the first embodiment in which this invention is applied to abnormal noise inspection in the final vehicle inspection process of automobiles. [Figure 2] A flowchart showing the processing flow of the first embodiment. [Figure 3] An explanatory diagram showing an example of the display in the display unit of the first embodiment. [Figure 4] Functional block diagram of the second embodiment. [Figure 5] A flowchart showing the processing flow of the second embodiment. [Figure 6] An explanatory diagram showing an example of the display in the display unit of the second embodiment. [Figure 7] An explanatory diagram of the enlarged image used for switching between wiper sounds. [Figure 8] An explanatory diagram showing other display examples. [Figure 9] Further explanatory diagrams showing other display examples. [Modes for carrying out the invention]

[0013] The following describes an embodiment of this invention applied to abnormal noise inspection in the final vehicle inspection process of automobiles. Generally, in the final vehicle inspection process of an automobile production line, inspectors test drive the finished vehicle to be inspected on a free roller and inspect numerous items, including the engine, transmission, meters, horn, brakes, wipers, etc., according to a predetermined inspection sequence. The inspection device of this embodiment acquires sound from the vehicle without relying on timing signals from the vehicle, and uses the acquired sound to extract target sounds, determine whether or not they are abnormal sounds (also called abnormal noises), and displays these results along with video.

[0014] In one embodiment, the sounds to be inspected include the horn and the wipers. For the horn, the inspector sounds the horn by pressing the horn switch on the steering wheel, and the inspection determines whether the horn is functioning correctly based on the sound. Since abnormalities such as a different pitch or an odd tone of sound can occur due to a malfunction of the horn itself, an incorrect part number (horn), poor wiring contact, a malfunction of the horn switch, etc., the inspection is performed based on the actual sound of the horn. Similarly, for the wipers, the inspector checks whether any abnormal sounds are coming from, for example, the area around the wiper blade when the wipers are operated.

[0015] FIG. 1 shows a functional block diagram of the inspection apparatus according to the first embodiment. Note that this inspection apparatus is not an independent apparatus as a horn / wiper inspection apparatus, but is configured as a part of a completed vehicle inspection apparatus. The inspection apparatus according to the first embodiment includes a sound acquisition unit 1, a video acquisition unit 10, a target sound identification unit 20, a noise determination unit 30, a sound / video data generation unit 40, a sound / video data storage unit 50, a video display control unit 60, and a display unit 70 including a speaker.

[0016] The sound acquisition unit 1 includes a microphone that acquires sound generated from the vehicle to be inspected and converts it into an electrical signal, that is, sound data, and a recording unit that temporarily stores this sound data. The microphone is disposed outside the vehicle so as to be able to collect sound from the vehicle including sounds such as a horn and a wiper. The directivity, sampling frequency, frequency band, sensitivity characteristics, etc. of the microphone are selected according to the measurement object and the environment of the sound field, etc. Usually, a microphone having directivity toward the vehicle is used. Sound data obtained by localizing the sound source using a microphone array or the like and removing the sound that becomes noise may be obtained.

[0017] The video acquisition unit 10 captures the vehicle to be inspected in parallel with the acquisition of sound and converts it into video data (that is, moving image data), and is, for example, a video camera. As the video camera, in order to enable partial enlargement of the acquired image, a video camera having a large number of pixels such as the full HD standard (1920×1080 pixels) is preferable. The video camera serving as the video acquisition unit 10 may be one or a plurality of cameras that perform shooting in synchronization with each other. In one embodiment, one video camera that captures the vehicle to be inspected from above (more specifically, obliquely upward) from the front is used.

[0018] The target sound identification unit 20 extracts the target sound (e.g., a horn sound) from the continuous sound data acquired by the sound acquisition unit 1 in virtually real time, and cuts out the sound data containing the target sound. For example, if the object of inspection is a horn, the unit focuses on a frequency band of, for example, 1 kHz, which is a characteristic of the horn sound, and identifies the target sound based on the condition that its sound pressure is 100 dBA or more. Then, it generates sound data of, for example, a few seconds in length, including the sound before and after this horn sound.

[0019] The abnormal sound detection unit 30 determines whether the target sound contained in the generated sound data, which is several seconds long, is an abnormal sound. For example, it quantifies the sound data using frequency analysis methods such as FFT (Fast Fourier Transform) or wavelet analysis, and calculates the degree of abnormality from a normal sound.

[0020] For example, the data can be converted into a two-dimensional frequency response (spectrum) with frequency on the horizontal axis and sound pressure or power on the vertical axis using frequency analysis, or all extracted data can be converted into a frequency response all at once, or the data can be extracted at a predetermined data length (time window), and the frequency response can be calculated while overlapping and sliding the data over time. Furthermore, the frequency response can be adjusted to match the auditory sensitivity by multiplying it by A-weighting or C-weighting, which are frequency response characteristics of human hearing sensitivity. In addition, sound data can be quantified by processing such as a three-dimensional spectrogram consisting of time, frequency, and sound pressure, or Mel-cepstrum used in speech recognition.

[0021] After performing this processing, feature quantities that characterize normal and abnormal sound data, such as the XY coordinates of the FFT centroid, the envelope of the spectrum, and its slope angle, are defined, and the degree of abnormality is calculated.

[0022] As for the anomaly score, one can use the results of calculating statistical outliers (= anomaly score) by calculating the mean or variance of the numerical values ​​specified as features of the abnormal sound, or by vectorizing features of two or more dimensions and using cosine similarity, which is expressed as the magnitude of the angle between them.

[0023] Then, thresholds are set for determining whether these abnormalities are normal or abnormal, and the calculated abnormality level is used to quantitatively distinguish between normal and abnormal sounds based on whether or not it exceeds the threshold.

[0024] Other methods include, for example, linearly transforming multiple values ​​using multivariate analysis to convert the multivariate into a one-dimensional value using the value with the highest contribution (anomaly score Z = A × feature A + B × feature B + C × feature C...), and then setting a threshold to determine the anomaly score.

[0025] When the sound / video data generation unit 40 determines that a sound is abnormal in the abnormal sound detection unit 30, it saves the aforementioned sound data, which is several seconds long and includes the target sound (horn sound), in an appropriate format, such as a WAVE file. Then, it extracts video data to a similar length range as the sound data and saves it as a video file of a suitable format, such as an AVI file, which is several seconds long. These WAVE and AVI files are stored as linked data in the sound / video data storage unit 50, which serves as a database. For example, they are grouped into a single folder (or a single compressed file) and stored in the sound / video data storage unit 50. Additional information such as text files containing vehicle information, sound data analysis results (spectrum, spectrogram, etc.), and detailed text files of the abnormal sound detection results are added to this folder. Alternatively, information such as vehicle information and detection results may be added to the file name when saving.

[0026] This data allows for accurate verification of abnormal sound data after measurement. Furthermore, this data can be used as training and validation data in AI such as deep learning.

[0027] The video display control unit 60 generates images or videos to be displayed on the display unit 70 based on the results obtained as described above, and also generates sound signals to be emitted from the speaker attached to the display unit 70. The display unit 70 is a display means for displaying the generated images or videos to relevant parties such as inspectors driving the vehicle, manufacturing managers, and data scientists who utilize the data. For example, it can be composed of a liquid crystal display, an organic EL display, a tablet, an HMD (Head Mounted Display), a smartwatch, etc. It also includes a sound source, amplifier, speaker, etc. for emitting sound.

[0028] If the video display control unit 60 determines that there is at least an abnormal sound, it will respond to the playback request by playing sound data through the speaker, as described later, and will simultaneously display the time-series sound data, including an indicator synchronized with the playback, the analysis results of the sound data, and the playback image of the video data synchronized with the playback of the sound data, all on a single screen of the display unit 70. In particular, with respect to the video data, in order to make the operation or action that caused the sound more clearly visible, the unit will crop, enlarge or reduce, tilt and rotate a part of the video, and adjust or correct the image quality (hue, sharpness, contrast, brightness), etc. For example, if the sound is a horn sound, in order to make it visible that the inspector pressed the horn switch on the steering wheel, the unit will crop and enlarge the driver's seat area from the overhead view of the front of the vehicle acquired by the video acquisition unit 10, and create a display video with increased contrast and brightness to compensate for the darkness inside the vehicle. Furthermore, in consideration of the inspector's privacy, processing is applied to prevent identification of individuals, such as blurring the inspector's face or replacing the face with an avatar image. In one embodiment, the partially enlarged image is displayed together with an overall image showing the entire vehicle from above, and the enlarged area is displayed in the form of a speech bubble so that it can be easily identified within the overall image.

[0029] Figure 3 is an explanatory diagram showing an example of the display in the display unit 70 of the first embodiment. This display is generated using data from various files in a single folder stored in the sound / video data storage unit 50 described above. In this example, at the top of the display screen, "(1) Type of sound" is displayed, indicating that it is a horn sound, and the vehicle type and management number for identifying the vehicle are displayed (see reference numeral 70a). In the middle section, "(2) Sound pressure / frequency characteristics of the sound" is displayed, showing sound data including the target sound with the horizontal axis as time and the vertical axis as sound pressure (see reference numeral 70b), and the frequency characteristics, i.e., spectrum (see reference numeral 70c), obtained by frequency analysis of this data, side by side. Here, the display of the sound data (70b), which is time-series data, includes an indicator (see reference numeral 70b1) consisting of, for example, a red straight line for displaying the time, which moves horizontally in sync with playback, allowing the user to visually see which part of the sound data is being played during playback.

[0030] Furthermore, at the bottom of the screen, under the heading "(3) Video Display," the following are displayed: an operation panel section including play and rewind buttons (see reference numeral 70d), an overall image of the vehicle viewed from the front (see reference numeral 70e), and a magnified image of the driver's seat area superimposed in a speech bubble-like manner on the overall image (see reference numeral 70f). Note that in Figure 3, the inspector is shown as an illustration, but in reality, the photographed inspector is displayed as is, except for the face.

[0031] In the lower right corner of the screen, under "(4) Sound Judgment Results," the blowing time, sound pressure, abnormal volume, judgment threshold, judgment result, etc., are displayed in a table format as text data (see reference numeral 70g).

[0032] In the initial screen (for example, immediately after the inspection is completed), the video data (70e, 70f) is either a still image or not displayed. When the administrator or other person verifying the test operates the play button on the control panel (70d) via an input means such as a mouse, sound data lasting several seconds is played through the speaker. Simultaneously, the indicator (70b1) in the sound data (70b) displayed on the screen begins to move to the right side of the diagram, allowing the location of the sound playback to be visually confirmed. Furthermore, the overall image (70e) and the enlarged image of the driver's seat (70f) begin to play simultaneously. These images (70e, 70f) and sounds are played in sync with each other.

[0033] Furthermore, to allow inspectors to verify or confirm the results themselves, if an abnormal sound is detected, the system may be configured to automatically start playing the sound or video (70e, 70f) immediately after the inspection is completed.

[0034] As shown in Figure 3, the display and reproduced sound allow those conducting the verification to visually confirm that the sound was indeed an abnormal noise by listening to the actual horn sound, looking at the sound data (70b), spectrum (70c), and the "(4) Sound Judgment Results" table (70g). Furthermore, the driver's seat enlarged image (70f), which is a video played in sync with the sound, allows the inspector to visually confirm the operation of the horn switch, and to visually confirm the sound data (70b) and listen to the sound reproduced from the speaker while comparing it with the timing of the operation. For example, if a sound is mistakenly identified as a horn sound and judged as an abnormal noise at a time clearly different from the timing of the inspector's operation of the horn switch, it is easy to determine that this abnormal noise judgment is incorrect by playing a video showing the horn switch operation in sync. In addition, for example, if the sound pressure level of the horn sound is extremely low, it is possible to retrospectively verify whether the inspector actually operated the horn switch.

[0035] In the above embodiment, there is only one video camera acting as the video acquisition unit 10, and a portion of its image is cropped and enlarged, making synchronization with sound playback relatively easy. When using multiple video cameras, it is possible to add images, for example, taken from the side of the vehicle so that the inspector's hands can be seen.

[0036] Furthermore, although not shown in Figure 3, the control panel (70d) can be equipped with buttons for slow or frame-by-frame playback. When slow or frame-by-frame playback is requested by pressing these buttons, the system can be configured to play audio data from the speaker, display the time on the indicator (70b1), and play video data (70e, 70f) at a relatively slow speed while maintaining synchronization with each other. This makes it easier to confirm, for example, the timing of horn switch operation.

[0037] Figure 2 is a flowchart showing the processing flow of the inspection apparatus of the first embodiment described above. First, the sound acquisition unit 1's microphone collects sounds, including, for example, the sound of the horn, from the vehicle undergoing a test run on the free roller and acquires them as sound data (Step 1). In parallel with this, the video camera of the video acquisition unit 10 films the vehicle undergoing a test run on the free roller and acquires it as video data (Step 2).

[0038] Next, the process moves to step 3, where it is determined whether the target sound (a horn sound in this example) occurred based on the sound pressure characteristics and frequency characteristics of the acquired sound. If the target sound is detected, the process moves from step 3 to step 4, where the sound data and video data are extracted into segments of a few seconds in length, including the range before and after the target sound, and stored in the sound / video data storage unit 50 as linked data.

[0039] Next, the process moves from step 4 to step 5, where the extracted sound data is analyzed to determine whether it is a normal or abnormal sound. In step 6, a screen display as shown in Figure 3 is generated. Then, in step 7, a screen display as shown in Figure 3 is displayed. As mentioned above, when playback is requested by operating the control panel (70d) for post-verification, the playback of sound data from the speaker, the time display on the indicator (70b1), and the playback of video data (70e, 70f) are performed in sync with each other. If an abnormal sound is detected, playback may be started automatically. Note that if a normal sound is detected, the generation of a screen for post-verification including video data (70e, 70f) may be omitted.

[0040] Finally, in step 8, it is determined whether the inspector pressed the stop button to end the inspection. If the stop button has not been pressed, the process from steps 1 to 7 is repeated. For example, if an abnormality in the horn sound is indicated, the horn switch can be operated again to repeat the horn inspection. Once the stop button is pressed, the inspection is terminated. At the end of the inspection, a message to that effect is displayed on the display unit 70. In addition to the display, a buzzer or voice notification may also be given to indicate the end of the inspection.

[0041] Next, the inspection apparatus of the second embodiment will be described based on Figures 4 to 6. The following description will mainly focus on the differences from the first embodiment. The inspection apparatus of the second embodiment differs from the first embodiment in that it extracts multiple inspection target sounds (e.g., horn sound and wiper sound) during a series of inspections, performs abnormal sound detection, and switches the display for post-inspection verification of the results.

[0042] Figure 4 shows a functional block diagram of the inspection display device of the second embodiment. The inspection device of the second embodiment, like the first embodiment, includes a sound acquisition unit 1, a video acquisition unit 10, a target sound identification unit 20, an abnormal sound determination unit 30, a sound / video data generation unit 40, a sound / video data storage unit 50, a video display control unit 60, and a display unit 70 including a speaker, and further includes a priority determination unit 80.

[0043] In the second embodiment, while sound and video are continuously acquired by the sound acquisition unit 1 and the video acquisition unit 10, the inspector operates the horn switch and the wiper switch at any time. The target sound identification unit 20 extracts the target sound (horn sound and wiper sound) from the continuous sound data acquired by the sound acquisition unit 1 in virtually real time and cuts out sound data containing the target sound. For example, if it is determined to be a horn sound, it generates sound data of a length of, for example, several seconds, including the sound before and after it. Similarly, if it is determined to be a wiper sound (mainly a sound produced by the wiper blade), it generates sound data of a length of, for example, several seconds, including the sound before and after it.

[0044] The abnormal noise determination unit 30 determines whether the extracted horn sound and wiper sound are normal sounds or abnormal sounds by an appropriate method, similar to the first embodiment described above.

[0045] The sound / video data generation unit 40, similar to the first embodiment, saves the aforementioned sound data, which includes the target sound (horn sound or wiper sound) and is several seconds long, as a WAVE file when the abnormal sound detection unit 30 determines that it is an abnormal sound. Then, it extracts video data to a similar length range as the sound data and saves it as a video file of several seconds, such as an AVI file. These WAVE files and AVI files are stored as linked data in the sound / video data storage unit 50, which serves as a database. For example, they are grouped into a single folder (or a single compressed file) and stored in the sound / video data storage unit 50. Additional information such as text files containing vehicle information, sound data analysis results (spectrum, spectrogram, etc.), and detailed text files of the abnormal sound detection results are added to this folder. Alternatively, information such as vehicle information and detection results may be added to the folder name or file name. In the second embodiment, if multiple target sounds are abnormal sounds, they are treated as linked data and, for example, grouped into a single folder.

[0046] The priority determination unit 80 determines the priority of which of the multiple target sounds, for example, a horn sound and a wiper sound, should be prioritized for display on the display unit 70 (including playback of the sound from the speaker), or which should be prioritized. In one example, the priority is determined based on the following four criteria.

[0047] 1. The magnitude of the impact when using the parts as they are. 2. Severity of abnormality 3. Sound pressure and duration 4. Timing of abnormal noise occurrence These four criteria are prioritized in descending order of importance. In other words, if the magnitude of the impact when using the part without replacement differs, the one with the greater impact has higher priority. If the magnitude of the impact is the same, the next criterion, the magnitude of the abnormality, is compared. If the magnitude of the abnormality is also the same, the next criterion, the magnitude of the sound pressure and duration, is compared. If these are also the same, the next criterion, the abnormal noise detected earlier, has higher priority.

[0048] In the example of horn noise and wiper noise, the horn noise takes higher priority according to the first criterion. Horns are items that require inspection because their performance is regulated by law, and using them while they are malfunctioning is not permitted by law. On the other hand, wipers are a product quality issue, so they have a relatively lower priority.

[0049] The second criterion is based on the fact that a greater degree of abnormality is more likely to be considered a problematic abnormal sound. Similarly, the third criterion takes into account that a higher sound pressure or longer duration is more likely to be considered a problematic abnormal sound.

[0050] The video display control unit 60 generates a screen display that takes these priorities into consideration, and displays it via the display unit 70.

[0051] Figure 6 is an explanatory diagram showing an example of the display in the display unit 70 of the second embodiment. In this example, since the horn sound has a higher priority, information regarding the horn sound is displayed preferentially, and information regarding the wiper sound is displayed by switching the screen. Figure 6 shows the screen when information regarding the horn sound is displayed, and is basically similar to the display of the first embodiment, but the "(1) Sound type" column (see reference numeral 70a) at the top of the display screen lists both the horn sound and the wiper sound, indicating that it is a display of two different sounds. The label for "horn sound" is marked with the number 1 in a circle, and the label for "wiper sound" is marked with the number 2 in a circle. Furthermore, the section labeled "(2) Sound pressure / frequency characteristics" in the middle (see reference numeral 70b), the control panel section (reference numeral 70d) and enlarged image (see reference numeral 70f) in the section labeled "(3) Video display" at the bottom, and the section labeled "(4) Sound judgment result" in the lower right (see reference numeral 70g) are each marked with the number 1 in a circle, indicating that these relate to horn sounds.

[0052] In this display state, if the play button on the control panel (70d) is operated, as described in the first embodiment above, the acquired horn sound (sound determined to be an abnormal sound) will be played back through the speaker, and the indicator (70b1) in the sound data (70b) displayed on the screen will start moving to the right side of the figure. At the same time, playback of the overall image (70e) and the enlarged image of the driver's seat (70f) will begin. These images (70e, 70f) and sounds will be played back in sync with each other.

[0053] On the other hand, the overall image of the vehicle viewed from the front (see reference numeral 70e) includes a circled number 2 with an arrow to indicate the location of the wiper sound. For example, if a person conducting a post-incident review clicks on the circled number 2 in this overall image (70e) or the circled number 2 in the "(1) Type of Sound" column (70a) using an input device such as a mouse, the display will switch to one related to the wiper sound.

[0054] In other words, the sound data containing the target sound (70b), the frequency-analyzed spectrum (70c), and the "(4) Sound Judgment Results" table (70g) are all switched to display data related to the wiper sound, and the circled number 2 is added to indicate that it is a wiper sound.

[0055] Furthermore, the enlarged image of the driver's seat in Figure 6 (70f) is switched to an enlarged image of the wiper area on the windshield (70f2), as shown in Figure 7. At this time, the number 1 in a circle and an arrow is displayed in the driver's seat area of ​​the overall image (70e), indicating that the display related to the horn sound is hidden.

[0056] In this display state, operating the play button on the control panel (70d) will play the acquired wiper sound (the sound determined to be an abnormal sound) through the speaker, and the indicator (70b1) in the sound data (70b) displayed on the screen will start moving to the right side of the figure. Simultaneously, playback of the overall image (70e) and the magnified image of the wiper area (70f2) will begin. These images (70e, 70f2) and sounds are played in sync with each other. The magnified image of the wiper area (70f2) is also a cropped portion of the original video data acquired via the video camera, with appropriate processing applied, showing the back-and-forth motion of the wiper that causes the wiper sound as a video. Therefore, for example, by comparing the wiper movement with the sound data (70b) displayed together with the played sound and indicator (70b1), it is possible to verify at what timing the abnormal wiper sound occurred, or in what manner the abnormal sound occurred.

[0057] Additionally, it may be possible to automatically start playing the sound or video each time the display switches between the horn sound and the wiper sound.

[0058] Figure 5 is a flowchart showing the processing flow of the inspection apparatus of the second embodiment. First, the sound acquisition unit 1's microphone collects sounds, including, for example, the sound of the horn, from the vehicle undergoing a test run on the free roller and acquires them as sound data (Step 1). In parallel with this, the video camera of the video acquisition unit 10 films the vehicle undergoing a test run on the free roller and acquires it as video data (Step 2).

[0059] Next, the process moves to step 3, where it is determined whether the target sound (in this example, a horn sound or a wiper sound) occurred based on the sound pressure characteristics and frequency characteristics of the acquired sound. If the target sound is detected, the process moves from step 3 to step 4, where audio and video data are extracted into segments of approximately a few seconds in length, including the range before and after the target sound, and stored in the audio / video data storage unit 50 as linked data. If multiple target sounds (horn sound and wiper sound) are detected during the series of inspections, audio and video data segments of approximately a few seconds are extracted for each of them.

[0060] Then, moving from step 4 to step 5, the extracted sound data is analyzed to determine whether each sound is normal or abnormal.

[0061] Next, in step 11, it is determined whether there were two or more types of abnormal sounds. If there were two or more types of sounds, the process proceeds to step 12, where the display priority is determined using the method described above.

[0062] Next, in step 6, screen displays as shown in Figure 6 are generated according to priority. Then, in step 7, the screen display as shown in Figure 6 is performed.

[0063] Finally, in step 8, it is determined whether the inspector pressed the stop button to end the inspection. If the stop button has not been pressed, the process from steps 1 to 7 is repeated. Once the stop button is pressed, the inspection is terminated.

[0064] Figure 8 is an explanatory diagram showing different examples of displays in the display unit 70 of the second embodiment. In this example, the display related to the horn sound shown in Figure (a) and the display related to the wiper sound shown in Figure (b) are displayed alternately at regular intervals. For example, the display related to the higher-priority horn sound is displayed for 10 seconds, and then the display related to the wiper sound is displayed for 10 seconds. It is preferable to repeat this multiple times. If the playback button on the operation panel (70d) is operated while either display is being shown, the respective abnormal sound will be played back, and the indicator (70d1) on the sound data (70d) synchronized with it will be displayed, as well as the overall image (70e) and enlarged images (70f, 70f2) will be played back.

[0065] Additionally, the system may be configured to automatically start playing sound and video each time the display is switched at regular intervals.

[0066] Figure 9 is an explanatory diagram showing yet another example of the display in the display unit 70 of the second embodiment. In this example, information related to the target sound with the highest priority (horn sound in this example) is basically displayed (sound data (70b), frequency-analyzed spectrum (70c), "(4) Sound Judgment Result" table (see reference numeral 70g in Figure 6), and enlarged image of the driver's seat (70f)). Also, similar to the example in Figure 6, the "(1) Sound Type" column at the top of the screen (see reference numeral 70a) lists both horn sound and wiper sound, with circled numbers 1 and 2 respectively. Furthermore, when information related to horn sound is displayed, as explained in Figure 6, the overall image (70e) of the vehicle viewed from the front is accompanied by a circled number 2 along with an arrow to indicate the location of the wiper sound.

[0067] If the play button on the control panel (70d) is pressed while this display is active, the acquired horn sound (sound determined to be an abnormal sound) will be played back through the speaker, the indicator (70b1) in the sound data (70b) displayed on the screen will start moving to the right side of the diagram, and simultaneously, the overall image (70e) and the enlarged image of the driver's seat (70f) will start playing back. These images (70e, 70f) and sounds will be played back in sync with each other.

[0068] Then, in this display state, clicking the circled number 2 in the overall image (70e) or the circled number 2 in the "(1) Sound Type" column (70a) using an input means such as a mouse will display an enlarged image (70f2) of the wiper portion on the windshield, as shown in Figure 7, in the area of ​​the "(4) Sound Judgment Result" table (see reference numeral 70g in Figure 6) as a display related to the wiper sound. For example, it may be displayed as a speech bubble blowing out from the wiper portion of the overall image (70e). At this time, a triangular play button (see reference numeral 70h) labeled "Sound Judgment Result Display" will also be displayed. Clicking this play button will make the enlarged image of the wiper portion (70f2) disappear and return to the display of the "(4) Sound Judgment Result" table.

[0069] When playback is initiated while the magnified image of the wiper area (70f2) is displayed, the wiper sound identified as an abnormal noise and the magnified image of the wiper area (70f2) will be played back in sync. Furthermore, the display of horn sound information, which has a higher priority, will continue to be shown in the sound data (70b) and spectrum (70c).

[0070] The above describes embodiments of applying this invention to the inspection of abnormal horn and wiper sounds. However, this invention is not limited to the above embodiments and can be applied to various other applications. For example, it can be applied to the detection of abnormal brake sounds in vehicles. For instance, video data including the operation of the brake caliper or brake operation by an inspector can be acquired with a video camera and played back in sync with the brake sound. It can also be used for sound inspections other than vehicle inspections. [Explanation of Symbols]

[0071] 1...Sound acquisition section 10…Video acquisition unit 20...Target sound identification section 30... Abnormal noise detection unit 40…Sound and video data generation unit 50…Audio / Video Data Storage Unit 60...Video display control unit 70… indicates the section 80…Priority Determination Department

Claims

1. The system acquires sound emitted from the equipment being tested and generates sound data that includes the target sound. Simultaneously with sound acquisition, video of the equipment under inspection is acquired, and video data including the operation or action that caused the sound to be generated is generated. The sound data is analyzed to determine whether the target sound is an abnormal sound. If at least an abnormal sound is detected, the system will respond to the playback request by playing the audio data through the speaker, and simultaneously display the time-series audio data, including an indicator synchronized with the playback, the analysis results of the audio data, and the playback image of the video data synchronized with the audio data playback, all on a single screen. Acoustic testing methods.

2. The acoustic inspection method according to claim 1, wherein sound data, video data, and analysis result data are stored in a database as interconnected data.

3. The acoustic inspection method according to claim 1, wherein the equipment to be inspected is a vehicle.

4. The sound inspection method according to claim 3, wherein the target sound is the sound of a vehicle's horn, and video data including the driver's horn operation is played back and displayed.

5. The sound inspection method according to claim 3, wherein the target sound is the sound of a vehicle's wiper operating, and video data including the operation of the wiper on the windshield is played back and displayed.

6. The acoustic inspection method according to claim 1, comprising generating video data by extracting an image of a target range from an image of one camera that has been temporarily recorded.

7. For multiple types of target sounds from the same device under inspection, the above sound data will be played back and displayed on the screen sequentially. The acoustic inspection method according to claim 1.

8. The system processes the playback and screen display of sound data for multiple types of target sounds according to a predetermined priority order. The acoustic inspection method according to claim 7.

9. When slow-motion playback is requested, the audio data playback, the time display by the indicator mentioned above, and the video data playback are all performed at a relatively slow speed while maintaining synchronization with each other. The acoustic inspection method according to claim 1.

10. A sound acquisition unit that acquires sound emitted from the equipment to be inspected, A video acquisition unit that acquires video of the equipment under inspection in parallel with sound acquisition, A sound and video data generation unit generates sound data including the target sound from acquired sound, and simultaneously generates video data including the operation or action that causes the generation of the target sound from acquired video in synchronization with this sound data. An abnormal sound detection unit analyzes sound data to determine whether the target sound is an abnormal sound, A display unit including a speaker, If it is determined that there is at least an abnormal sound, the display control unit will respond to the playback request by playing sound data through the speaker, and will display the time-series sound data including an indicator synchronized with the playback, the analysis results of the sound data, and the playback image of video data synchronized with the playback of the sound data, all on one screen of the display unit. An acoustic inspection device comprising the following:

Images