Fusing audio, visual and sensor context information in mobile capture

Abstract

The disclosed systems and methods include a context detection module that detects a current context of an environment of a mobile device. Audio and video processing of audio and images captured by a microphone and camera of the device, respectively, in the environment is determined based on the detected context. The context detection module contains at least one audio classifier and at least one visual classifier. In some embodiments, the context detection module can be extended to use sensor information, in place of or in addition to, the audio and visual information. The captured audio, visual and sensor information are aligned on a time axis based on outputs of the audio classifier, the visual classifier and timestamps associated with the sensor information. One or more fusion methods are used to combine the context detection results.

Description

FUSING AUDIO, VISUAL AND SENSOR CONTEXT INFORMATION IN MOBILE CAPTURECROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of priority from U.S. Provisional Patent Application No. 63 / 550,287, filed February 6, 2024, and PCT International Patent Application No.PCT / CN2023 / 102812, filed June 27, 2023, each of which is incorporated by reference herein in its entirety.TECHNICAL FIELD

[0002] The disclosed embodiments relate to audio, image and video processing, and in particular to combining audio, visual and sensor context information in mobile capture.BACKGROUND

[0003] User-generated content (UGC) is typically created by consumers and can include any form of content (e.g., images, videos, text, audio). One trend related to UGC is personal moment sharing in variable environments (e.g., indoors, outdoors, by the sea) by recording video and audio using a personal mobile device (e.g., smart phone, tablet computer, wearable device).SUMMARY

[0004] Various embodiments are disclosed for fusing audio, visual and sensor context information in mobile capture.

[0005] In some embodiments, a method comprises: receiving, with at least one processor of a mobile device, input signals including audio, video and sensor signals captured by the mobile device; extracting, with the at least one processor, an audio feature vector from the audio signal; extracting, with the at least one processor, a visual feature vector from the video signal; extracting, with the at least one processor, a sensor feature vector from the sensor signal; and generating, using a classifier, a classification decision indicating an environment context type based on the audio, visual and sensor feature vectors.

[0006] In some embodiments, the method further comprises segmenting, using a sliding window, the audio signal into overlapping segments and performing the extracting and generating steps on the overlapping segments.

[0007] In some embodiments, the audio, video and sensor signals are continuously captured by the mobile device.

[0008] In some embodiments, the method further comprises resampling the visual and sensor feature vectors to match a length of the audio feature vector.

[0009] In some embodiments, the method further comprises normalizing at least one of audio, visual or sensor feature vectors to limit the feature vectors to a specified range.

[0010] In some embodiments the method further comprises generating confidence scores for the first, second and third context detections.

[0011] In some embodiments, at least one audio feature in the audio feature vector is Mel-frequency cepstral coefficients (MFCC), at least one visual feature in the visual feature vector is a color model value and at least one feature in the sensor feature vector is a measure of ambient light of the environment.

[0012] In some embodiments, a method comprises: receiving, with at least one processor of a mobile device, input signals including audio, video and sensor signals captured in an operating environment of a mobile device; extracting, with the at least one processor, an audio feature vector from the audio signal; detecting, with the at least one processor, a first environment context type and first confidence score for the first environment context type with an audio classifier and based on the audio feature vector; extracting, with the at least one processor, a visual feature vector from the video signal; detecting, with the at least one processor, a second environment context type and second confidence score for the second environment context type with a visual classifier and based on the visual feature vector; extracting, with the at least one processor, a sensor feature vector from the sensor signal; detecting, with the at least one processor, a third environment context type and third confidence score for the third environment context type with a sensor classifier and based on the sensor feature vector; and generating a final classification decision of the environment context type based on the first, second and third confidence scores

[0013] In some embodiments, the first, second and third confidence scores are weighted according to the confidence scores.

[0014] In some embodiments, a weight for the second confidence score is reduced when a user of the mobile device zooms a camera of the mobile device into a specific object.

[0015] In some embodiments, a weight for the second confidence score is adjusted based on a quality of an image captured by a camera of the mobile device.

[0016] In some embodiments, the method further comprises determining that there is music playing on the mobile device and reducing a weight of the audio classifier.

[0017] In some embodiments, a weight for the second confidence score is adjusted based on a time of day.

[0018] In some embodiments, a weight for the first confidence score is adjusted if

[0019] a microphone of the device is occluded.

[0020] In some embodiments, a weight for the first confidence score is adjusted based on whether the input audio signal was edited.

[0021] In some embodiments, the method further comprises: determining, by a motion sensor of the device, that the device is shaking while the method is performed; purging the final classification decision of the environment context type; determining, by the motion sensor, that the device is no longer shaking; detecting new first, second and third environment context types and corresponding first, second, and third confidence scores; and generating a new final classification decision of the environment context type based on the new first, second, and third corresponding confidence scores.

[0022] In some embodiments, a weight for the second confidence score is reduced when an image captured by a camera of the mobile device is blurred.

[0023] In some embodiments, the visual classifier estimates if the mobile device is indoors or outdoors.

[0024] In some embodiments, the weights are time varying.

[0025] In some embodiments, at least one of the weights is adjusted when a change in the environment is detected.

[0026] In some embodiments, the first weight for the first confidence score is reduced.

[0027] In some embodiments, a method comprises: receiving, with at least one processor of a mobile device, input signals including audio, video and sensor signals captured in an operating environment of a mobile device; extracting, with the at least one processor, an audio feature vector from the audio signal; detecting, with the at least one processor, a first environment context type and first confidence score for the first environment context type with an audio classifier and based on the audio feature vector; generating a first classification decision of the environment context type based on the first confidence score; extracting, with the at least one processor, a visual feature vector from the video signal; detecting, with the at least one processor, a second environment context type and second confidence score for the second environment context type using a visual classifier and based on the visual feature vector; generating a second classification decision of the environment context type based on the second confidence score; extracting, with the at least one processor, a sensor feature vector from the sensor signal; detecting, with the at least one processor, a third environment contexttype and third confidence score for the third environment context type with a sensor classifier and based on the sensor feature vector; generating a third classification decision of the environment context type based on the third confidence score; generating a final classification decision of the environment context type based on a majority among the first, second and third classification decisions.

[0028] In some embodiments, if two of the three classification decisions are the same, one of the two classification decisions is determined based on confidence score for the two classification decisions or accuracy on a validation dataset.

[0029] In some embodiments, a computer program includes instructions which, when executed by a computing apparatus, cause the computing apparatus to perform any of the preceding methods.

[0030] In some embodiments, a non-transitory computer-readable storage medium storing instructions which, when executed by a computing apparatus, cause the computing apparatus to perform any of the preceding methods.

[0031] In some embodiments, a computing apparatus comprises: a display; at least one processor; and memory storing instructions, which when executed by the at least one processor, cause the computing apparatus to perform any of the preceding methods.

[0032] The disclosed embodiments provide at least one or more of the following advantages. The disclosed embodiments can be used for realistically capturing an environment or scene for UGC while maintaining the creative objective of the content creator.BRIEF DESCRIPTION OF THE DRAWINGS

[0033] For a better understanding of the various described embodiments, reference should be made to the Detailed Description below, in conjunction with the following drawings in which like reference numerals refer to corresponding parts throughout the figures.

[0034] FIG. 1 is a block diagram of a feature-level fusion method, in accordance with one or more embodiments.

[0035] FIG. 2 is a block diagram of a score-level fusion method in accordance with one or more embodiments.

[0036] FIG. 3 is a block diagram of an alternative score-level fusion method in accordance with one or more embodiments.

[0037] FIG. 4 is a flow diagram of a feature-level fusion method, in accordance with one or more embodiments.

[0038] FIG. 5 is a flow diagram of a score-level fusion method in accordance with one or more embodiments.

[0039] FIG. 6 is a flow diagram of an alternative score-level fusion method in accordance with one or more embodiments.

[0040] FIG. 7 is a block diagram of a system for implementing the features and processes described in reference to FIGS. 1-6, according to some embodiments.DETAILED DESCRIPTION

[0041] Fusion methods are disclosed that combine audio and visual context information in a mobile capture and saves the audio and visual context information to metadata. More particularly, the disclosed method includes using a context detection module to detect a current environment context type (e.g., indoors, outdoors). Audio and video processing of audio and images captured by a microphone and camera, respectively, in the environment is determined based on the detected context. The context detection module contains at least one audio classifier and at least one visual classifier. In some embodiments, the context detection module can be extended to use sensor information (e.g., a lux value measuring light intensity) in addition to the audio and visual information. The captured audio, visual and sensor information are aligned on a time axis based on outputs of the audio classifier, the visual classifier and timestamps associated with the sensor information. One or more fusion methods described below are used to combine the context detection results from audio, visual and sensor processing channels to generate a final decision on environment context type using, for example, a voting scheme.

[0042] In some embodiments, the detection module classifies the environment to one of several environment context types and also detects changes to that environment, e.g., detecting transitions between indoor and outdoor environments. For each environment context type, a different audio and video processing model can be applied. For example, for audio processing, a different mixing ratio of environment sound and target event sound can be set, the target loudness of an event sound can be set to a different loudness target, and the amount of reverberation or width of the environment can be set. For video processing, different camera parameters can be set to improve the image capture quality, e.g., to make images captured in an indoor environment “warmer” than images captured in an outdoor environment. In some embodiments, different audio and video processing profiles based on context detection results can be performed in real time and / or offline. Metadata can be obtained from the context detection results and stored for editing or other applications in, e.g., an offline mode.Example Inter-path Fusion Methods

[0043] Inter-path fusion methods are disclosed for combining context information from one or more processing paths, such as audio, visual and sensor processing paths. The interpath fusion methods expect one or more prediction vectors as input and outputs a single prediction vector. In some embodiments, a final class is obtained using a maximum membership rule. In some embodiments, context information resulting from different methods may be partially or entirely recorded in metadata for editing or other applications in, for example, an offline mode.

[0044] Two embodiments are disclosed below for fusing audio, visual, and optionally sensor context information: feature-level fusion and score-level fusion. Feature-level fusion obtains features from audio, visual, and optionally sensor processing paths, and uses the features to generate a classification decision on environment context type. Score-level fusion obtains context information from audio, visual, and optionally sensor paths and uses a fusion module to combine the context information to generate a final classification decision on environment context type.Feature-level Fusion

[0045] FIG. 1 is a block diagram of a feature-level fusion method 100, in accordance with one or more embodiments. As illustrated in FIG. 1 , audio features 101 and visual features 102 are extracted from an input signal (e.g., a video signal with audio) separately in separate audio and visual processing channels, respectively. In some embodiments, a sliding window is used to continuously segment the audio signal as the window slides through the audio signal. The percentage of overlapping of segments can be pre-defined. The segmentation of the audio signal allows the audio processing path to recognize context information in the audio signal adaptively and continuously and output a continuous set of audio feature vectors 101 to fusion model 104.

[0046] Similarly, for the visual processing path, image frames are continuously captured and processed to recognize context information in the image frames and extract visual feature vectors 102 that are output to fusion model 104. Feature vectors 101, 102 from the audio and visual paths, respectively, are extracted for a specified length of the input signal (e.g., a video sequence).

[0047] In some embodiments, an optional sensor processing path extracts a sensor feature vector 103 from sensor data that is continuously capture by device sensors. The visualand sensor feature vectors 102, 103 can be resampled to match the length of the audio feature vector 101 if the frame rate is diverse for different content, and the sensor information will be diverse on different devices. This embodiment allows for the extraction of matched audio, visual, and sensor feature vectors over a continuous video sequence. The features can be used to train a context classifier by combining corresponding label information context as the target. In some embodiments, normalization can be used to force the input feature vectors into a reasonable range of values for the classifier. Finally, feature vectors 101, 102, 103 from the audio, visual, and sensor processing paths, respectively, will correspond to a feature vector of confidence scores.

[0048] In some embodiments, Mel-frequency cepstral coefficients (MFCCs) is an example audio feature included in audio feature vector 101, an RGB frame is an example visual feature included in visual feature vector 102 and a lux value from an ambient light sensor of the device is an example sensor feature included in sensor feature vector 103. Any other audio, visual or sensor features can be included in feature vectors 101, 102 and 103.

[0049] Fusion model 104 takes the audio, visual and sensor feature vectors as input and outputs a classification decision and optionally a confidence score for the classification decision. Fusion model 104 is discussed more fully below.Score-level Fusion

[0050] For score- level fusion, there are three options. For option 1, features 201-203 are input into classifiers 203-206 which output class predictions and corresponding confidence scores for each class prediction. In some embodiments, classifiers 203-206 can be implemented as described in U.S. Patent Application No. 18 / 548,791, which bears the title “Context Aware Soundscape Control”, filed on September 1, 2023, and which is incorporated by reference herein in it is entirety. The confidence scores are weighted by weights 207-209 and added together to generate final classification decision 210, as shown in FIG. 2. In some embodiments, final classification decision 210 is obtained from the highest score among the weighted classification scores. The weights can either be trained or selected according to statistics. The weights can further improve the accuracy of the final classification decision 210 by reducing the impact of poor performing classifiers 203-206 and increasing the contribution of the well -performing classifiers 203-206, as determined by their corresponding confidence scores.

[0051] For option 2, the class decision of each processing channel is used in a majority voting scheme, as shown in FIG. 3. Feature vectors 201-203 are input into classifiers 204-206,respectively, which output classification decisions for audio, visual and sensors. The outputs of the classifiers 204-206 are input into majority voting module 211 which outputs a final classification decision based on the number of occurrences (i.e., “votes”) for each classification decision to generate the final classification decision. Unlike option 1, option 2 uses the predicted labels obtained with a maximum membership rule. The predicted label that occurs the most is selected as the final classification decision. If there are multiple predicted labels with the same number of occurrences (i.e., same number of votes), one predicted label can be selected based on other rules like, for example, a confidence score or accuracy on a validation dataset. Following this method, every classifier’s class probability can also be given a performance weight.

[0052] In option 3, the weights 207-209 for each confidence score can be time varying. For example, the analysis window of audio classifier 204 can be reset or given less weight on historical classification decisions during a scene switch that can be determined by, for example, detecting a change in the environment based on visual cues (e.g., moving from indoors to outdoors or vice versa).Fusion Method of Indoor / Qutdoor Classifier in Specific Cases

[0053] There are several specific cases where the prediction decision from audio and visual path will be continuously different and not just a difference between confidence scores. These specific cases are discussed below.Scene Changes Suddenly (e.g., Passing Through Door)

[0054] In this case, if the previous detection result is indoor, the visual classifier may change to outdoor if the user passes through a door from the outside of a building, for example. In this case, the final classification decision may change only when the sensor information (e.g., lux level) has a greater change than a specified threshold and the confidence score change of the audio classifier is also greater than a specified threshold.Capture Something Through a Window

[0055] In this case, the class decision of the visual classifier and audio classifier will be different consistently at axis. For example, when you are capturing the landscape though the window on a train, the class decision of the audio classifier will be indoor, but the class decision of the visual classifier will be outdoor. In this case, all the audio related processing can bebased on the result from the audio classifier and all the video related processing will be based on the result from visual classifier.Capture Something Where Reverberation is Heavy in the Outdoor Scene

[0056] In this case, the audio classifier will be less accurate, and the result from visual classifier will be trusted more, and at least set to a lower weight on the result from the audio channel when making a final classification decision. In this embodiment, the fusion process can use a speech, noise, and music classifier.Music case

[0057] In this case, the audio classifier may be easily influenced by loud music, so the result from visual classifier will be trusted more and at least set a lower weight on the result from the audio path when making the final classification decision.

[0058] In some cases, it is hard to distinguish between a scene change and capturing something through a window. Therefore, in an offline mode, a further fusion method can be used to combine audio and visual cues to help distinguish between these two cases. Video processing typically relies on visual cues while audio processing typically relies on audio, except during a scene switch, and at the beginning of capture where visual features could be leveraged to switch faster.

[0059] To handle the scene switch case, the fusion method can use video cues to perform segmentation on audio results and apply a majority voting scheme on each audio segment. If there’ s only a visual scene switch (e.g. , capture indoors and rotate the camera to outside through a window), the audio voting would be same for two consecutive segments. If it is case 1 (“through the door”), the audio voting would be different as better results can be achieved by resetting the audio analysis window. To handle the beginning part of capture where the prediction results from the audio classifier may be less accurate, the visual classifier can be relied upon at the beginning of the capture.Extension: Sensor-Information Fusion

[0060] In some embodiments, sensor information is used to adaptively modify the running interval of audio and visual classifiers. Specifically, if the sensor information indicates there is no scene change, for the audio path, the audio feature is calculated consistently and the audio classifier is disabled. The decision of previous result is copied to save computation. The visual path is processed in a similar manner to the audio path. When sensor informationindicates there is probably a scene change, audio and visual classifiers are used to capture the change where the audio features stored in a buffer are used to run the audio classifier, if necessary, since most of the audio classifier works on feature vectors obtained within a window / segment, that is it requires features of history frames.

[0061] Specifically, a lux value from the light sensor and Global Navigation Satellite System (GNSS) coordinates can be used for scene switch detection. An accelerometer sensor is helpful to detect blur images caused by obvious camera shaking which may result in a less accurate visual analysis results. Tntra-path weighting can be used with this blur information, as described below.Intra-path Fusion Within Each Channel

[0062] In addition to fusing the context information of multiple channels, the accuracy of the corresponding results for each channel by voting within a given window can be improved. This avoids the impact of some outliers in the detection results. For example, when a microphone of the device is subjected to an occlusion the audio classifier results may not be accurate. To address this, when performing the voting process in the window, the weight of the microphone during the occlusion period can be reduced. Similarly, when visual information is used to make judgments about the environment context (e.g., indoor, outdoor, or other scenes), the rapid movement of the user will lead to blurring of the video. For example, when the user zooms their camera into a specific object, the likelihood of the video containing environmental information is reduced. Therefore, the weight of the visual classifier results during this zooming period can be reduced to improve robustness.

[0063] In some embodiments, a voting scheme within a window is denoted as:[1] where i and j are the frame indexes, conf is a vector which contains the confidence score of each scene (e.g., indoor: 0.6, outdoor: 0.4), confvotingis the voting result of all the conf vectors in a window W (mostly contains current frame and several history frames) and aLis the weight of confidence of each frame. Then the final confidence can be obtained by normalizing confvoting. If the proposed intra-path fusion method is not applied, cq = 1. When there is a scene switch, a buffer of history conf vectors are flushed and the current conf vector is be appended to the voting window.

[0064] In some embodiments, the intra-fusion method for the visual path uses the Laplacian variance of an image to detect whether the image is blurred. A weight is then set to be aL< 1 for a blurred image input. For example, the threshold blurTHof determining whether it is a blurred input is set, the weight atcan be obtained by:[2] where co and 9 are scaling parameters that map the weight a, range to [0,1]. For example, the scaling parameters can be set as c = 0.9, 9 = 0.1 to linearly reduce the weight a, when the Laplacian variance of the input is lower.

[0065] Additionally, the dominant colors of the input image can be determined, and if the input frame contains less colors than a threshold number (e.g., 3 colors), and the maximum proportion of one color is higher than a threshold (e.g., 0.92), a very small weight aLcan be set for the current frame.Metadata Format

[0066] In some embodiments, the audio and visual metadata are stored separately to steer audio and video processing respectively. For each modality, there could be separate metadata for different aspects of that modality. For example, for audio the metadata can indicate indoor / outdoor, speech / music, and for visual the metadata can indicate indoor / outdoor, lux, etc. In some embodiments, metadata can be combined and used to control the same set of processing parameters in the same way. However, for indoor / outdoor fusion of different modality, the same aspect is addressed from a different modality. For both audio and visual metadata, the results can be at a low sample rate (e.g., 1 result / second). However, for scene switch cases a sub-second resolution can be used.Time Alignment of Context Information in Real Time Mode

[0067] In real time, multi context information alignment is used to determine the environment context at the beginning of capture and when the environment context changes, i.e., scene change. In some embodiments, the video classifier estimates the current environment context type (e.g., indoor or outdoor) from the beginning of the capture. The audio classifier estimates the current environment context type based on the date of the recorded audio. Other sensors (e.g., the ambient light sensor), can be used to estimate thepossibility of scene change by calculating the average change rate of the ambient light of the environment.

[0068] Since there is no history data recorded at the beginning of capture, the video classifier output can be used to determine the current environment context type. Meanwhile, the audio classifier and other sensors (e.g., the ambient light sensor) output the estimated types at certain time intervals (e.g., 1 output per-second).

[0069] When the output score of audio or video jitter occurs, the current context type can be updated to the new context type using the vote-based fusion method described above. When the value of the light sensor changes greater than a threshold, the current context type refers to the video output score. When the video and light sensor are inactive values, the current context type refers to the audio output score and updates the new context type using the vote-based fusion method once the video and light sensor’ s values are restored to a certain range.Example Processes

[0070] FIG. 4 is a flow diagram of a feature-level fusion method, in accordance with one or more embodiments. An example method comprises: receiving input signals including audio, video and sensor signals captured by the mobile device (401), extracting an audio feature vector from the audio signal (402), extracting a visual feature vector from the video signal (403), extracting a sensor feature vector from the sensor signals (404), and generating, using a classifier, a classification decision indicating an environment context type based on the audio, visual and sensor feature vectors (405). Each of these steps was previously described in detail in reference to FIG. 1.

[0071] FIG. 5 is a flow diagram of a score-level fusion method in accordance with one or more embodiments. An example method comprises: receiving input signals including audio, video and sensor signals captured in an operating environment of a mobile device (501), extracting an audio feature vector from the audio signal (502), detecting a first environment context type and first confidence score for the first environment context type with an audio classifier and based on the audio feature vector (503), extracting a visual feature vector from the video signal (504), detecting a second environment context type and second confidence score for the second environment context type with a visual classifier and based on the visual feature vector (505), extracting a sensor feature vector from the sensor signal (506), detecting a third environment context type and third confidence score for the third environment context type with a sensor classifier and based on the sensor feature vector (507), and generating a final classification decision of the environment context type based on the first, second and thirdconfidence scores (508). Each of these steps was previously described in detail in reference toFIG. 2.

[0072] FIG. 6 is a flow diagram of an alternative score-level fusion method in accordance with one or more embodiments. An example method comprises: receiving input signals including audio, video and sensor signals captured in an operating environment of a mobile device (601); extracting an audio feature vector from the audio signal (602); detecting a first environment context type and first confidence score for the first environment context type with an audio classifier and based on the audio feature vector (603); generating a first classification decision of the environment context type based on the first confidence score (604); extracting a visual feature vector from the video signal (605); detecting a second environment context type and second confidence score for the second environment context type with a visual classifier and based on the visual feature vector (606); generating a second classification decision of the environment context type based on the second confidence score (607); extracting a sensor feature vector from the sensor signal (608); detecting third environment context type and third confidence score for the third environment context type with a sensor classifier and based on the sensor feature vector (609); generating a third classification decision of the environment context type based on the third confidence score (610); generating a final classification decision of the environment context type based on a majority among the first, second and third classification decisions (611). Each of these steps was previously described in detail in reference to FIG. 3.

[0073] FIG. 7 shows a block diagram of an example system for implementing the features and processes described in reference to FIGS. 1-6, according to an embodiment. System 700 includes any devices that are capable of playing audio, including but not limited to: smart phones, tablet computers, wearable computers, vehicle computers, game consoles, surround systems, kiosks.

[0074] As shown, the system 700 includes a central processing unit (CPU) 701 which is capable of performing various processes in accordance with a program stored in, for example, a read only memory (ROM) 702 or a program loaded from, for example, a storage unit 708 to a random access memory (RAM) 703. In the RAM 703, the data required when the CPU 701 performs the various processes is also stored, as required. The CPU 701, the ROM 702 and the RAM 703 are connected to one another via a bus 709. An input / output (I / O) interface 705 is also connected to the bus 704.

[0075] The following components are connected to the I / O interface 705: an input unit 706, that may include a keyboard, a mouse, or the like; an output unit 707 that may include adisplay such as a liquid crystal display (LCD) and one or more speakers; the storage unit 708 including a hard disk, or another suitable storage device; and a communication unit 709 including a network interface card such as a network card (e.g., wired or wireless).

[0076] In some implementations, the input unit 706 includes one or more microphones in different positions (depending on the host device) enabling capture of audio signals in various formats (e.g., mono, stereo, spatial, immersive, and other suitable formats).

[0077] In some implementations, the output unit 707 include systems with various number of speakers. As illustrated in FIG. 7, the output unit 707 (depending on the capabilities of the host device) can render audio signals in various formats (e.g., mono, stereo, immersive, binaural, and other suitable formats).

[0078] The communication unit 709 is configured to communicate with other devices (e.g., via a network). A drive 710 is also connected to the I / O interface 705, as required. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a flash drive or another suitable removable medium is mounted on the drive 710, so that a computer program read therefrom is installed into the storage unit 708, as required. A person skilled in the art would understand that although the system 700 is described as including the above-described components, in real applications, it is possible to add, remove, and / or replace some of these components and all these modifications or alteration all fall within the scope of the present disclosure.

[0079] In accordance with example embodiments of the present disclosure, the processes described above may be implemented as computer software programs or on a computer- readable storage medium. For example, embodiments of the present disclosure include a computer program product including a computer program tangibly embodied on a machine readable medium, the computer program including program code for performing methods. In such embodiments, the computer program may be downloaded and mounted from the network via the communication unit 709, and / or installed from the removable medium 711, as shown in FIG. 7.

[0080] The foregoing description, for purpose of explanation, has been described with reference to specific embodiments. However, the illustrative discussions above are not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations are possible in view of the above teachings. The embodiments were chosen and described in order to best explain the principles of the techniques and their practical applications. Others skilled in the art are thereby enabled to best utilize the techniques andvarious embodiments with various modifications as are suited to the particular use contemplated.

[0081] Although the disclosure and examples have been fully described with reference to the accompanying drawings, it is to be noted that various changes and modifications will become apparent to those skilled in the art. Such changes and modifications are to be understood as being included within the scope of the disclosure and examples as defined by the claims.

[0082] Various aspects of the disclosure may be appreciated from the following Enumerated Example Embodiments (EEEs):

[0083] EEE 1. A method comprising: receiving, with at least one processor of a mobile device, input signals including audio, video and sensor signals captured by the mobile device; extracting, with the at least one processor, an audio feature vector from the audio signal; extracting, with the at least one processor, a visual feature vector from the video signal; extracting, with the at least one processor, a sensor feature vector from the sensor signal; and generating, with a classifier, a classification decision indicating an environment context type based on the audio, visual and sensor feature vectors.

[0084] EEE 2. The method of EEE 1, further comprising: segmenting, using a sliding window, the audio signal into overlapping segments and performing the extracting and generating steps on the overlapping segments.

[0085] EEE 3. The method of EEE 1 or 2, wherein the audio, video and sensor signals are continuously captured by the mobile device.

[0086] EEE 4. The method of any preceding EEE, further comprising resampling the visual and sensor feature vectors to match a length of the audio feature vector.

[0087] EEE 5. The method of any preceding EEE, further comprising normalizing at least one of the audio, visual or sensor feature vectors to limit the feature vectors to a specified range.

[0088] EEE 6. The method of any preceding EEE, further comprising generating a confidence score for the classification decision.

[0089] EEE 7. The method of any preceding EEE, where at least one audio feature in the audio feature vector is Mel-frequency cepstral coefficients (MFCC), at least one visual feature in the visual feature vector is a color model value and at least one feature in the sensor feature vector is a measure of ambient light of the environment.

[0090] EEE 8. A method comprising: receiving, with at least one processor of a mobile device, input signals including audio, video and sensor signals captured in an operating environment of a mobile device; extracting, with the at least one processor, an audio feature vector from the audio signal; detecting, with the at least one processor, a first environment context type and first confidence score for the first environment context type with an audio classifier and based on the audio feature vector; extracting, with the at least one processor, a visual feature vector from the video signal; detecting, with the at least one processor, a second environment context type and second confidence score for the second environment context type with a visual classifier and based on the visual feature vector; extracting, with the at least one processor, a sensor feature vector from the sensor signal; detecting, with the at least one processor, a third environment context type and third confidence score for the third environment context type with a sensor classifier and based on the sensor feature vector; and generating a final classification decision of the environment context type based on the first, second and third confidence scores.

[0091] EEE 9. The method of EEE 8, wherein the first, second and third confidence scores are weighted according to the confidence scores.

[0092] EEE 10. The method of EEE 9, wherein a weight for the second confidence score is reduced when a user of the mobile device zooms a camera of the mobile device into a specific object.

[0093] EEE 11. The method of EEE 9 or EEE 10, wherein a weight for the second confidence score is adjusted based on a quality of an image captured by a camera of the mobile device.

[0094] EEE 12. The method of any of EEEs 9 to 11, further comprising: determining that there is music playing on the mobile device; and reducing a weight of the audio classifier.

[0095] EEE 13. The method of any of EEEs 9 to 12, wherein a weight for the second confidence score is adjusted based on a time of day.

[0096] EEE 14. The method of any of EEEs 9 to 13, wherein a weight for the first confidence score is adjusted if a microphone of the device is occluded.

[0097] EEE 15. The method of any of EEEs 9 to 14, wherein a weight for the first confidence score is adjusted based on whether the input audio signal was edited.

[0098] EEE 16. The method of any of EEEs 9 to 15, further comprising: determining, by a motion sensor of the device, that the device is shaking while the method is performed; purging the final classification decision of the environment context type; determining, by the motion sensor, that the device is no longer shaking; detecting new first, second and thirdenvironment context types and corresponding first, second, and third confidence scores; and generating a new final classification decision of the environment context type based on the new first, second, and third corresponding confidence scores.

[0099] EEE 17. The method of any of EEEs 9 to 16, wherein the visual classifier estimates if the mobile device is indoors or outdoors.[000100] EEE 18. The method of EEE 9 or any claim dependent thereon, wherein the weights are time varying.[000101] EEE 19. The method of EEE 9 or any claim dependent thereon, wherein at least one of the weights is adjusted when a change in the environment is detected.[000102] EEE 20. The method of EEE 9 or any claim dependent thereon, wherein the first weight for the first confidence score is reduced.[000103] EEE 21. A method comprising: receiving, with at least one processor of a mobile device, input signals including audio, video and sensor signals captured in an operating environment of a mobile device; extracting, with the at least one processor, an audio feature vector from the audio signal; detecting, with the at least one processor, a first environment context type and first confidence score for the first environment context type with an audio classifier and based on the audio feature vector; generating a first classification decision of the environment context type based on the first confidence score; extracting, with the at least one processor, a visual feature vector from the video signal; detecting, with the at least one processor, a second environment context type and second confidence score for the second environment context type with a visual classifier and based on the visual feature vector; generating a second classification decision of the environment context type based on the second confidence score; extracting, with the at least one processor, a sensor feature vector from the sensor signal; detecting, with the at least one processor, a third environment context type and third confidence score for the third environment context type with a sensor classifier and based on the sensor feature vector; generating a third classification decision of the environment context type based on the third confidence score; generating a final classification decision of the environment context type based on a majority vote among the first, second and third classification decisions.[000104] EEE 22. The method of EEE 21, wherein if two of the three classification decisions are the same, one of the two classification decisions is determined based on confidence score for the two classification decisions or accuracy on a validation dataset.[000105] EEE 23. A computer program including instructions which, when executed by a computing apparatus, cause the computing apparatus to perform the method of any of EEEs 1- 22.[000106] EEE 24. A non-transitory computer-readable storage medium storing instructions which, when executed by a computing apparatus, cause the computing apparatus to perform the method of any of EEEs 1 to 22.[000107] EEE 25. A computing apparatus, comprising: a display; at least one processor; and memory storing instructions, which when executed by the at least one processor, cause the computing apparatus to perform the method of any of EEEs 1 to 22.

Claims

CLAIMSWhat is claimed is:

1. A method comprising: receiving, with at least one processor of a mobile device, input signals including audio, video and sensor signals captured by the mobile device; extracting, with the at least one processor, an audio feature vector from the audio signal; extracting, with the at least one processor, a visual feature vector from the video signal; extracting, with the at least one processor, a sensor feature vector from the sensor signal; and generating, with a classifier, a classification decision indicating an environment context type based on the audio, visual and sensor feature vectors.

2. The method of claim 1 , further comprising: segmenting, using a sliding window, the audio signal into overlapping segments and performing the extracting and generating steps on the overlapping segments.

3. The method of claim 1 or 2, wherein the audio, video and sensor signals are continuously captured by the mobile device.

4. The method of any preceding claim, further comprising resampling the visual and sensor feature vectors to match a length of the audio feature vector.

5. The method of any preceding claim, further comprising normalizing at least one of the audio, visual or sensor feature vectors to limit the feature vectors to a specified range.

6. The method of any preceding claim, further comprising generating a confidence score for the classification decision.

7. The method of any preceding claim, where at least one audio feature in the audio feature vector is Mel-frequency cepstral coefficients (MFCC), at least one visual feature in the visualfeature vector is a color model value and at least one feature in the sensor feature vector is a measure of ambient light of the environment.

8. A method comprising: receiving, with at least one processor of a mobile device, input signals including audio, video and sensor signals captured in an operating environment of a mobile device; extracting, with the at least one processor, an audio feature vector from the audio signal; detecting, with the at least one processor, a first environment context type and first confidence score for the first environment context type with an audio classifier and based on the audio feature vector; extracting, with the at least one processor, a visual feature vector from the video signal; detecting, with the at least one processor, a second environment context type and second confidence score for the second environment context type with a visual classifier and based on the visual feature vector; extracting, with the at least one processor, a sensor feature vector from the sensor signal; detecting, with the at least one processor, a third environment context type and third confidence score for the third environment context type with a sensor classifier and based on the sensor feature vector; and generating a final classification decision of the environment context type based on the first, second and third confidence scores.

9. The method of claim 8, wherein the first, second and third confidence scores are weighted according to the confidence scores.

10. The method of claim 9, wherein a weight for the second confidence score is reduced when a user of the mobile device zooms a camera of the mobile device into a specific object.

11. The method of claim 9 or 10, wherein a weight for the second confidence score is adjusted based on a quality of an image captured by a camera of the mobile device.

12. The method of any one of claims 9 to 11, further comprising:determining that there is music playing on the mobile device; and reducing a weight of the audio classifier.

13. The method of any one of claims 9 to 12, wherein a weight for the second confidence score is adjusted based on a time of day.

14. The method of any one of claims 9 to 13, wherein a weight for the first confidence score is adjusted if a microphone of the device is occluded.

15. The method of any one of claims 9 to 14, wherein a weight for the first confidence score is adjusted based on whether the input audio signal was edited.

16. The method of any one of claims 9 to 15, further comprising: determining, by a motion sensor of the device, that the device is shaking while the method is performed; purging the final classification decision of the environment context type; determining, by the motion sensor, that the device is no longer shaking; detecting new first, second and third environment context types and corresponding first, second, and third confidence scores; and generating a new final classification decision of the environment context type based on the new first, second, and third corresponding confidence scores.

17. The method of any one of claims 9 to 16, wherein the visual classifier estimates if the mobile device is indoors or outdoors.

18. The method of claim 9 or any claim dependent thereon, wherein the weights are time varying.

19. The method of claim 9 or any claim dependent thereon, wherein at least one of the weights is adjusted when a change in the environment is detected.

20. The method of claim 9 or any claim dependent thereon, wherein the first weight for the first confidence score is reduced.

21. A method comprising: receiving, with at least one processor of a mobile device, input signals including audio, video and sensor signals captured in an operating environment of a mobile device; extracting, with the at least one processor, an audio feature vector from the audio signal; detecting, with the at least one processor, a first environment context type and first confidence score for the first environment context type with an audio classifier and based on the audio feature vector; generating a first classification decision of the environment context type based on the first confidence score; extracting, with the at least one processor, a visual feature vector from the video signal; detecting, with the at least one processor, a second environment context type and second confidence score for the second environment context type with a visual classifier and based on the visual feature vector; generating a second classification decision of the environment context type based on the second confidence score; extracting, with the at least one processor, a sensor feature vector from the sensor signal; detecting, with the at least one processor, a third environment context type and third confidence score for the third environment context type with a sensor classifier and based on the sensor feature vector; generating a third classification decision of the environment context type based on the third confidence score; generating a final classification decision of the environment context type based on a majority vote among the first, second and third classification decisions.

22. The method of claim 21, wherein if two of the three classification decisions are the same, one of the two classification decisions is determined based on confidence score for the two classification decisions or accuracy on a validation dataset.

23. A computer program including instructions which, when executed by a computing apparatus, cause the computing apparatus to perform the method of any of claims 1-22.

24. A non-transitory computer-readable storage medium storing instructions which, when executed by a computing apparatus, cause the computing apparatus to perform the method of any of claims 22.

25. A computing apparatus, comprising: a display; at least one processor; and memory storing instructions, which when executed by the at least one processor, cause the computing apparatus to perform the method of any of claims 1 -22.

Images