Method for recording multimedia files and electronic device thereof

Abstract

The present disclosure provides a method for recording multimedia files using an electronic device. The method includes: previewing a scene including multiple objects in the field of view of a camera of the electronic device; and detecting at least one of a shape event and a sound event associated with at least one of the multiple objects in the scene. In addition, the method includes: determining at least one of a recording mode and an effect of at least one object based on at least one of the sound event and the shape event; and applying at least one of the recording mode and the effect. The method also includes: recording a multimedia file including at least one object in at least one of a recording mode video and an effect; and storing the multimedia file.

Description

Technical Field

[0001] The present disclosure relates to performing actions in an electronic device. More particularly, the present disclosure relates to an electronic device and method for recording a multimedia file including at least one object in at least one of a recording mode and an effect. Background Art

[0002] Generally, electronic devices are widely used to capture and view multimedia files such as videos. However, when a user captures a video using an electronic device, the user must manually apply the desired effect to a specific portion of the video because the process of capturing the video is not automatic and the desired effect cannot be determined creatively. Therefore, the process of manually applying the desired effect each time the user captures a video makes the process cumbersome. In addition, the effect applied by the user depends largely on the user's creativity and may not be an appropriate effect for a specific portion of the video. The mode switching mechanism itself is not sufficient to achieve automation or intelligence. For example, consider a scenario where a user is capturing a video of a person jumping into a swimming pool. The user will want to capture the dive in slow motion mode as well as a zoom effect to focus on the person who is diving. However, when the user applies the slow motion mode and the zoom effect, there is a possibility that the person may have already jumped and the user missed capturing a portion of the diving action.

[0003] The above information is presented only as background information to help the reader understand the present disclosure. Applicant has not made any determination, nor assertion, as to whether any of the above may be applied as prior art with respect to the present application. Summary of the invention

[0004] Technical Solution

[0005] The present disclosure provides a method and an electronic device for recording multimedia files. The method includes: previewing a scene including multiple objects in the field of view of a camera of the electronic device; and detecting at least one of a shape event and a sound event associated with at least one of the multiple objects in the scene. In addition, the method includes: determining at least one of a recording mode and an effect of the at least one object based on at least one of the sound event and the shape event; and automatically applying the recording mode and at least one of the effects. The method also includes: recording a multimedia file including the at least one object in at least one of the recording mode video and the effect; and storing the multimedia file.

[0006] The present disclosure also provides a method and electronic device for recording multimedia files. The method includes recording a first part of a scene in at least one of a first recording mode and a first effect, and while recording the scene, detecting at least one of a shape event and a sound event associated with the at least one object. In addition, the method also includes determining a second recording mode based on the shape event and the sound event, and recording the second part of the scene in at least one of the second recording mode and a second effect. The method also includes generating a multimedia file including the first part of the scene and the second part of the scene, and storing the multimedia file.

[0007] The present disclosure will also be better understood and appreciated when considered in conjunction with the following description and the accompanying drawings. However, it should be understood that the following description, although indicating preferred embodiments and numerous specific details thereof, is given by way of example and not limitation. Many changes and modifications may be made within the scope of the embodiments herein without departing from the spirit of the present disclosure, and the embodiments herein include all such modifications. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The present disclosure is illustrated in the accompanying drawings, and the same reference numerals throughout the drawings indicate corresponding parts in the various figures. The embodiments of the present invention will be better understood through the following description with reference to the accompanying drawings, in which:

[0009] Figure 1A is a block diagram of an electronic device for recording a multimedia file including at least one object in at least one of a recording mode and an effect according to an embodiment disclosed herein;

[0010] Figure 1B illustrates the functions of various elements of a recording mode management engine when determining at least one object and recording a multimedia file after applying at least one of a recording mode and an effect according to an embodiment disclosed herein;

[0011] Figure 1C is a block diagram of an object detection engine 131 for detecting at least one object in a scene according to embodiments disclosed herein;

[0012] Figure 1D is a block diagram of a frame prediction engine 132 of a recording mode management engine 130 according to an embodiment disclosed herein;

[0013] Figure 1E is an example showing a method of detecting a shape event by the shape event detection engine 133 according to an embodiment disclosed herein;

[0014] Figure 2Ais a model of a frame rate determination engine 135 for determining a frame rate based on at least one of a shape event and a sound event associated with at least one object according to embodiments disclosed herein;

[0015] Figure 2B-2E is a diagram showing recording modes and effects applied to multimedia files based on frame rates according to embodiments disclosed herein;

[0016] Figure 3 is a flow chart illustrating a method for recording a multimedia file including at least one object in at least one of a recording mode and an effect according to an embodiment disclosed herein;

[0017] Figure 4 is a flow chart illustrating a method for generating and playing back a multimedia file including a first portion of a scene and a second portion of the scene according to embodiments disclosed herein;

[0018] Figure 5A-5D are examples of various techniques for displaying a unique identifier corresponding to at least one of a recording mode and an effect of a multimedia file according to an embodiment disclosed herein;

[0019] Figure 6 is an example illustration of detecting a shape event and a sound event associated with a balloon and recording a multimedia file in at least one of a recording mode and an effect when no secondary object is present within a field of view (FOV) of a camera according to an embodiment disclosed herein;

[0020] Figure 7 is an example illustration for detecting shape events and sound events associated with a bow and recording a multimedia file in at least one of a recording mode and an effect according to embodiments disclosed herein;

[0021] Figure 8 is an example illustration for detecting a shape event associated with a dolphin and recording a multimedia file in at least one of a recording mode and an effect according to an embodiment disclosed herein;

[0022] Fig. 9 is an example illustration for detecting a shape event associated with a balloon and recording a multimedia file in at least one of a recording mode and an effect when a secondary object is present in a FOV of a camera of an electronic device according to an embodiment disclosed herein;

[0023] Fig.10 is an example illustration of detecting a shape event associated with a basket in a basketball court and recording a multimedia file in at least one of a recording mode and an effect according to embodiments disclosed herein;

[0024] Fig.11is an example showing selecting a target and automatically recording abnormal actions performed by the target at a higher frame rate according to an embodiment disclosed herein;

[0025] Fig.12 is an example illustration of detecting a shape event associated with a balloon and recording a multimedia file in at least one of a recording mode and an effect when a secondary object is present within a FOV of a camera of a wearable device according to an embodiment disclosed herein;

[0026] Fig.13 is an example illustration for detecting shape events and sound events associated with a vehicle and recording a multimedia file in at least one of a recording mode and an effect according to an embodiment disclosed herein; and

[0027] Fig.14 is an example illustration for capturing a plurality of objects by recording a plurality of multimedia files in at least one of a recording mode and an effect using a plurality of cameras, and generating a multimedia file by merging the plurality of multimedia files according to an embodiment disclosed herein. DETAILED DESCRIPTION

[0028] Various embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. In the following description, only specific details such as detailed configuration and components are provided to help understand these embodiments of the present disclosure as a whole. Therefore, it is obvious to those skilled in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. In addition, for the sake of clarity and brevity, the description of well-known functions and configurations is omitted. Moreover, the various embodiments described herein do not have to be mutually exclusive, because some embodiments can be combined with one or more other embodiments to form new embodiments. In this article, unless otherwise stated, the term "or" used herein refers to a non-exclusive "or". The examples used herein are only intended to facilitate understanding of the way in which the embodiments herein can be practiced, and further enable those skilled in the art to practice the embodiments herein. Therefore, the examples should not be interpreted as limiting the scope of the embodiments herein.

[0029] As tradition in the art, embodiments can be described and illustrated according to the block of performing one or more functions described.These blocks may be referred to as units, engines, managers, modules, etc. in this article, and are physically performed by analog and / or digital circuits (such as logic gates), integrated circuits, microprocessors, microcontrollers, storage circuits, passive electronic components, active electronic components, optical components, hard-wired circuits, etc., and may be optionally driven by firmware and / or software.Circuit may be, for example, implemented in one or more semiconductor chips, or implemented on substrate supports (such as printed circuit boards, etc.).The circuit constituting the block may be realized by dedicated hardware, or by a processor (for example, one or more programmed microprocessors and associated circuits), or by a combination of dedicated hardware performing some functions of the block and a processor performing other functions of the block.Without departing from the scope of the present disclosure, each block of the embodiment may be physically divided into two or more interacting and discrete blocks.Equally, without departing from the scope of the present disclosure, the block of the embodiment may be physically combined into more complex blocks.

[0030] Therefore, embodiments of the present invention provide a method for recording multimedia files using an electronic device. The method includes: previewing a scene including multiple objects in the field of view of a camera of the electronic device; and detecting at least one of a shape event and a sound event associated with at least one of the multiple objects in the scene. In addition, the method includes: determining at least one of a recording mode and an effect for at least one object based on at least one of the sound event and the shape event; and automatically applying at least one of the recording mode and the effect. The method also includes recording a multimedia file including at least one object in at least one of a recording mode video and an effect; and storing the multimedia file.

[0031] In conventional methods and systems, the selection of recording modes is a completely user-driven process, where the user must individually set the desired recording mode and must repeat the setting when needed, and real-time recording functions are not possible. Unlike conventional methods and systems, the proposed method is fully automatic and determines the recording mode and / or effect to be applied to the video based on shape events and / or sound events associated with at least one object in the scene.

[0032] Unlike conventional methods and systems, the proposed method detects contextual possible objects that may appear in the scene at a later point in time, and therefore, the processing time required to determine the recording mode / effect becomes faster.

[0033] Referring now to the drawings, and more particularly to Figures 1A to 13 , wherein like reference numerals indicate corresponding features throughout the several views, there is shown a preferred embodiment.

[0034] Figure 1A 1 is a block diagram of an electronic device 100 for recording a multimedia file including at least one object in at least one of a recording mode and an effect according to an embodiment disclosed herein. Figure 1A As shown, the electronic device 100 may be, for example, a laptop computer, a personal computer, a mobile phone, a smart phone, a personal digital assistant (PDA), a tablet computer, a wearable device, etc. In an embodiment, the electronic device 100 may include a camera 110, a voice management engine 120, a recording mode management engine 130, a processor 140, a memory 150, and a display 160.

[0035] In an embodiment, the camera 110 is configured to preview a scene including multiple objects within a field of view. In addition, the camera 110 is further configured to record a multimedia file including at least one object in at least one of a recording mode and an effect. The camera 110 is capable of capturing multimedia files at a variable frame rate. In an embodiment, the voice management engine 120 includes a microphone 122 and a voice detection engine 124. The microphone 122 in the voice management engine 120 is configured to receive voice input from a user for selecting / providing a secondary object in the form of a voice command. The voice detection engine 124 in the voice management engine 120 is also configured to determine a sound associated with at least one object and provide it to the recording mode management engine 130, which in turn determines whether a sound event has occurred. In an embodiment, the recording mode management engine 130 is configured to detect at least one of a shape event and a sound event associated with at least one of the multiple objects in the scene. The shape event can be a sudden increase or decrease in the shape of the at least one object. For example, a tiger in a jungle suddenly sees prey and starts chasing the prey. The shape of the tiger suddenly changes. A sound event may be a sudden increase or decrease in the amplitude of a sound from at least one object. For example, a bullet is fired from a gun. A glass bottle suddenly falling and breaking is an example of a shape event and a sound event. At least one object in the scene may be a primary object and a secondary object. In addition, the recording mode management engine 130 is configured to determine at least one of a recording mode and an effect of at least one object based on at least one of the sound event and the shape event, and automatically apply at least one of the recording mode and the effect. The recording mode may be, for example, a slow motion mode, a fast motion mode, a panoramic mode, etc. The effect may be, for example, focusing, defocusing, etc. In addition, the recording mode management engine 130 is configured to record a multimedia file including at least one object in at least one of the recording mode and the effect.

[0036] In another embodiment, the recording mode management engine 130 is configured to record a first portion of a scene in at least one of a first recording mode and a first effect, and detect at least one of a shape event and a sound event associated with at least one object while recording the scene. In addition, the recording mode management engine 130 is configured to determine a second recording mode based on the shape event and the sound event, and record the second portion of the scene in at least one of the second recording mode and the second effect. The recording mode management engine 130 is also configured to generate a multimedia file including the first portion of the scene and the second portion of the scene. The first recording mode and the second recording mode may be at least one of a fast motion mode and a slow motion mode. The recording mode management engine 130 also embeds a unique identifier corresponding to the first recording mode on a timeline of the multimedia file while recording the first portion of the scene, and embeds a unique identifier corresponding to the second recording mode on a timeline of the multimedia file while recording the second portion of the scene.

[0037] In an embodiment, the processor 140 is configured to interact with hardware elements such as the camera 110 , the voice management engine 120 , the recording mode management engine 130 , the memory 150 , and the display 160 .

[0038] In an embodiment, the memory 150 is configured to store a multimedia file including at least one object recorded in at least one of the recording mode video and effects. The memory 150 also includes an object database that stores key-value pairs of object names and sample images. When the electronic device 100 detects multiple objects in the scene, the memory 150 also stores the main object selected by the user. In addition, the memory 150 also stores secondary objects determined based on the context of the main object. The memory 150 may include a non-volatile storage element. Examples of such non-volatile storage elements may include a magnetic hard disk, an optical disk, a floppy disk, a flash memory, or an electrically programmable memory (EPROM) or an electrically erasable programmable (EEPROM) memory. In addition, in some examples, the memory 150 may be considered to be a non-transitory storage medium. The term "non-transitory" may indicate that the storage medium is not embodied in a carrier or a propagated signal. However, the term "non-transitory" should not be interpreted as the memory 150 being non-removable. In some examples, the memory 150 is configured to store a larger amount of information than the memory. In some examples, non-transitory storage media may store data that can change over time (eg, in random access memory (RAM) or cache).

[0039] In an embodiment, the display 160 is configured to display a scene including a plurality of objects and receive touch input from a user for selecting an object, starting / ending recording, etc. In addition, the display 160 is configured to display detected objects and preview and record frames. In addition, the display 160 is configured to display a unique identifier corresponding to a first recording mode on a timeline of a multimedia file while recording a first portion of a scene, and to display a unique identifier corresponding to a second recording mode on a timeline of a multimedia file while recording a second portion of a scene. In addition, when a user plays back a stored multimedia file, the display 160 automatically displays a unique identifier corresponding to the first recording mode and the second recording mode on a timeline of the multimedia file.

[0040] Figure 1B Functions of various elements of the recording mode management engine 130 are illustrated when determining at least one object and recording a multimedia file after applying at least one of a recording mode and an effect according to an embodiment disclosed herein.

[0041] In an embodiment, the recording mode management engine 130 includes an object detection engine 131, a frame prediction engine 132, a shape event detection engine 133, a sound event detection engine 134, and a frame rate determination engine 135. In an embodiment, the object detection engine 131 is configured to detect related objects among multiple objects while previewing a scene. In addition, the object detection engine 131 uses a combination of a convolutional neural network (CNN) and contextual information to determine objects in a scene. The object detection engine 131 unit obtains an input frame and determines possible objects present in the scene, thereby reducing multiple processing levels. In an embodiment, the frame prediction engine 132 is a deep predictive coding network for video prediction. The frame prediction engine 132 is configured to receive an input frame, perform learning on the input frame, and predict future frames. The predicted future frames are sent to the object detection engine 131 to determine the contextually possible objects in the input frame. In an embodiment, the shape event detection engine 133 is configured to detect the shape change of the selected object and determine the shape change rate of the object. In an embodiment, the sound event detection engine 134 is configured to determine the sound pressure level (SPL) of the sound associated with at least one object, and determine whether the SPL of the sound associated with the at least one object meets the sound change criteria. When the sound associated with the at least one object meets the sound change criteria, the sound event is detected. The SPL change with respect to time can be re-expressed as the ratio r(t) of the linear ratio between the current signal power and the earlier signal power, as shown in the following equation:

[0042]

[0043] The current signal power pow(t) refers to the current signal power at T wThe average signal power during the period (dead time between pulses) is expressed as:

[0044]

[0045] p(t) is the SPL measured from microphone 122 .

[0046] Since the level of background noise changes slowly over time or in the environment, background noise is also included in the measured sound pressure. When only background noise is present, small changes in r(t) with respect to time can be observed. When a sudden loud sound is generated, r(t) changes dramatically, so a threshold value r is introduced. thr To distinguish whether the change of r(t) with respect to time is caused by a sudden loud sound. Therefore, the sound event is detected using equation (3), where the sudden sound d(t) is defined as:

[0047]

[0048] In an embodiment, the frame rate determination engine 135 is configured to combine the outputs of the shape event detection engine 133 and the sound event detection engine 134 from the previously trained regression model and return a predicted frame rate. If the predicted frame rate is different from the normal frame rate of the primary object, a focus effect is applied to the object and the frame rate is changed. Similarly, if the predicted frame rate is different from the normal rate of the secondary object, only the frame rate is changed. In addition, based on the previous occurrence state of the sound event or shape event, when the sound event or shape event disappears, the state switches to normal.

[0049] Figure 1C is a block diagram of an object detection engine 131 for detecting at least one object in a scene according to embodiments disclosed herein.

[0050] The proposed method uses a CNN-based model with contextual information to improve object detection and improve the accuracy of detecting at least one object in a scene. In addition, the method uses supervised learning. The method of object detection includes locating object instances in a scene (hypothesis generation) and classifying objects into semantic classes (hypothesis classification). Hypotheses are generated using features such as symmetry, aspect ratio, expected position, color, and motion. Generally, hypothesis classification methods can be divided into shape-based methods and feature-based methods. The proposed method is a feature-based method. The object detection engine 131 has a fusion system that merges the output from a CNN-based deep learning (DL) classifier and the output of a context-based classifier. The context-based classifier is designed using a naive Bayesian (Bayes) method. A Bayesian model is used to fuse DL classifier scores and context-based classifier scores. The object detection engine 131 is extensively trained using input frames to detect context objects in subsequent frames.

[0051] Figure 1D is a block diagram of the frame prediction engine 132 of the recording mode management engine 130 according to an embodiment disclosed herein.

[0052] The frame prediction engine 132 is a deep predictive coding network for video prediction, which is initially trained. The frame prediction engine 132 uses unsupervised learning. The frame prediction engine 132 learns to predict future frames sequentially in a multimedia file. Each layer within the frame prediction engine 132 makes local predictions and only forwards deviations from these predictions to subsequent layers. The predicted frames are sent to the object detection engine 131 for object detection to find possible objects from the predicted frames.

[0053] refer to Figure 1D , which shows a diagram of the information flow within each layer of the frame prediction engine 132. Each layer is represented by neurons (R 1 ), which is composed of a neuron at each time step (A ^ 1 ) outputs layer-specific predictions and compares the layer-specific predictions to the target (A 1 ) to generate the error term (E 1 ), which is then propagated horizontally and vertically in the network.

[0054] Figure 1E 2 is an example showing a method of detecting a shape event by the shape event detection engine 133 according to an embodiment disclosed herein.

[0055] The shape event detection engine 133 includes a framework for performing shape event detection on a sequence of 2-D silhouettes of deformable objects, which can be obtained from a sequence of frames by background subtraction and centroid alignment. A silhouette sequence is a video sequence where each pixel is either 0 (black) for background or 1 (white) for foreground (typically a moving / deformed object). When an action clip is received, the shape event detection engine 133 extracts a set of 13-dimensional shapes that describe the shape over time t. 0 How to change the geometry of the timeline, such as Figure 1E As shown. In addition, for the calculated values ​​of the 10 spatiotemporal distances of the object experiencing the shape event, the total change of the shape of the object in each frame is determined. If the difference between the sums of the 10 spatiotemporal distances is greater than a certain tuning threshold, the shape event is detected. In addition, the shape change rate is calculated by means of the amount of shape change experienced by the object over a period of time.

[0056] Figure 2A is a model of the frame rate determination engine 135 for determining a frame rate based on at least one of a shape event and a sound event associated with at least one object according to embodiments disclosed herein.

[0057] Figure 2B-2E is a diagram showing recording modes and effects applied to multimedia files based on frame rates according to embodiments disclosed herein.

[0058] refer to Figure 2A , the frame rate determination engine 135 includes a regression model such as linear regression to predict the frame rate. The frame rate determination engine 135 is trained by a feature set that includes input sound loudness, shape change rate, and expected output (frame rate). In addition, the frame rate determination engine 135 learns to predict the frame rate based on the variables of the sudden input sound and shape change rate using equation (4):

[0059] Y=β 0 +β 1 X 1 +β 2 X 2 (4)

[0060] Where: X 1 =Sudden sound, X 2 = shape change rate, Y = expected frame rate, β 1 =X 1 The coefficient, β 2 =X 2 The coefficient, β 0 =intercept.

[0061] For example, consider the main object in the scene, the value is X 1 =20,X 2 =30,β 1 =1,β 2 =1,β 0 =24. Therefore, substituting the values ​​in equation (4), (F 1 )Y=24+20+30=74fps,F final1 =(F+F 1 )=24+74=98fps.

[0062] In another example, consider the main object detected in the scene, value X 1 =40,X 2 =20,β 1 =1,β 2 =1,β 0 =24. Therefore, substituting the values ​​in equation (4), (F 2 )Y=24+40+20=84fps.

[0063] Because F 2 ≥F 1 , so F 2 =F1 / 2=74 / 2=35fps. So F final2 =(F+F2)=24+35=59fps. Assume that the normal frame rate is F (usually 24fps, 30fps).

[0064] From the curve (4), it can be concluded that in the main object (F 1 ). If the shape / sound event is gradual (slow shape change), a negative value is derived from the curve, whereas when the shape / sound event is sudden (sudden sound / +high shape change rate), a positive value is derived. The final frame rate of the video recording the shape / sound event with the main object is then F final =(F+F 1 ). If F final1 Different from the normal frame rate F, a signal is sent to the camera to focus on the object and change the frame rate to F final1 Therefore, when the main object experiences a shape / sound event, the frame rate (F1*) is increased and a focusing effect is provided, such as Figure 2B shown.

[0065] If an interesting event is detected in a secondary object, a frame rate (F 2 ). The value is normalized wrt,

[0066]

[0067] Then, the final frame rate of the secondary object event is F final2 =(F+F 2 ). If F final2 If the frame rate is different from the normal frame rate F, a signal is sent to change the frame rate to F. final2 Based on the final stage of sending an event-based intent, the state switches between normal frame rate and defocus. Figure 2C As shown, when the secondary object experiences a shape / sound event, the frame rate is increased (F2*).

[0068] When no sudden shape / sound events are detected or when shape / sound events occur gradually, the frame rate (F1' and F2'*) is reduced, and the reduction is for all objects in the scene, such as Figure 2D When the sound disappears and / or the shape changes stop, normal recording mode is resumed. Figure 2E The combined values ​​of shape events and sound events are shown in , which are obtained by the regression model in the frame rate determination engine 135.

[0069] Figure 33 is a flowchart 300 illustrating a method for recording a multimedia file including at least one object in at least one of a recording mode and an effect according to embodiments disclosed herein.

[0070] refer to Figure 3 In step 302, the electronic device 100 previews a scene including a plurality of objects in the field of view of the camera 110. Figure 1A In the electronic device 100 shown in FIG. 1 , the camera 110 is configured to preview a scene including a plurality of objects in a field of view of the camera 110 .

[0071] In step 304, the electronic device 100 determines the sound pressure level (SPL) of the sound associated with at least one object. In step 306, the electronic device 100 determines whether the SPL of the sound associated with at least one object meets the sound standard. In step 308, when it is determined that the SPL of the sound associated with at least one object meets the sound standard, the electronic device 100 detects at least one of the shape event and the sound event associated with the at least one object. In step 310, the electronic device 100 determines at least one of the recording mode and the effect of at least one object based on at least one of the sound event and the shape event. In step 312, the electronic device 100 automatically applies at least one of the recording mode and the effect. In step 314, the electronic device 100 records a multimedia file including at least one object with at least one of the recording mode and the effect, and stores the multimedia file. In step 316, the electronic device 100 obtains a plurality of frames associated with at least one object. In step 318, the electronic device 100 determines the difference of at least one region of at least one object in each frame. In step 320, the electronic device 100 determines the change rate of at least one region of at least one object in each frame. In step 322, the electronic device 100 determines whether the rate of change of at least one region of at least one object in each frame satisfies the shape criterion. In addition, when it is determined that the rate of change of at least one region of at least one object in each frame satisfies the shape criterion, the electronic device 100 loops to step 310 and executes steps 310 to 314 as described above. Figure 1A The steps from step 304 to step 322 are shown to be performed by the recording mode management engine 130 of the electronic device 100.

[0072] Figure 4 is a flow chart 400 illustrating a method for generating and playing back a multimedia file including a first portion of a scene and a second portion of the scene according to embodiments disclosed herein.

[0073] refer to Figure 4In step 402, the electronic device 100 records a first portion of the scene in at least one of a first recording mode and a first effect. In step 404, the electronic device 100 detects at least one of a shape event and a sound event associated with at least one object while recording the scene. In step 406, the electronic device 100 determines a second recording mode based on the shape event and the sound event. In step 408, the electronic device 100 records a second portion of the scene in at least one of a second recording mode and a second effect. In step 410, the electronic device 100 generates a multimedia file including the first portion of the scene and the second portion of the scene. In step 412, the electronic device 100 stores the multimedia file. For example, in Figure 1A In the illustrated electronic device 100 , the memory 150 is configured to store multimedia files.

[0074] In step 414, the electronic device 100 plays back the multimedia file. In step 416, the electronic device 100 automatically displays the unique identifier corresponding to the recording mode on the timeline of the multimedia file. Figure 1A In the electronic device 100 shown, the display 160 is configured to automatically display a unique identifier corresponding to the recording mode on the timeline of the multimedia file. In step 418, the electronic device 100 plays back the first part in the first recording mode and plays back the second part in the second recording mode. Figure 1A The recording mode management engine 130 of the electronic device 100 is shown to execute steps from step 402 to step 410 , step 414 , and step 418 .

[0075] Figure 5A-5D are examples of various techniques of displaying a unique identifier corresponding to at least one of a recording mode and an effect of a multimedia file according to an embodiment disclosed herein.

[0076] refer to Figure 5A, at 502a, the primary object in the frame is a bow. The electronic device 100 determines a shape event in the primary object. In addition, based on the shape event in the primary object, the electronic device 100 determines at least one of a recording mode and an effect to be applied while recording a multimedia file including the primary object. In addition, the recording mode is changed by increasing the frame rate, and a focus effect is applied to highlight the primary object on the screen. The increase in the frame rate of the multimedia file is indicated by providing a unique identifier (such as a color change of the timeline) compared to the normal frame rate of the multimedia file. In addition, a representation of the primary object is also provided on the timeline of the multimedia file. At 504a, a secondary object is captured when a sound event is detected. In addition, based on the sound event caused by the secondary object, the electronic device 100 determines at least one of a recording mode and an effect to be applied while recording a multimedia file including the secondary object. In addition, the recording mode is changed by increasing the frame rate to highlight the secondary object on the screen compared to the normal frame rate of the multimedia file. The increase in the frame rate of the multimedia file is indicated by providing a unique identifier (such as a color change of the timeline) compared to the normal frame rate of the multimedia file and the frame rate provided for the primary object of the multimedia file. In addition, a representation of secondary objects is also provided on the timeline of the multimedia file. Figure 5A ,exist Figure 5B In the embodiment of the present invention, at 502b, the increase in the frame rate of the multimedia file is indicated by providing a unique identifier (such as a graphical representation with a color change of the timeline) compared to the normal frame rate of the multimedia file. At 504b, the increase in the frame rate of the multimedia file is indicated by providing a unique identifier (such as a graphical representation with a color change of the timeline) compared to the normal frame rate of the multimedia file and the frame rate provided for the primary object of the multimedia file.

[0077] Combination Figure 5A ,exist Figure 5C In the embodiment, at 502c, the increase in the frame rate of the multimedia file is indicated by providing a unique identifier (such as a zigzag timeline with a color change) compared to the normal frame rate of the multimedia file. At 504c, the increase in the frame rate of the multimedia file is indicated by providing a unique identifier (such as a zigzag timeline with a color change) compared to the normal frame rate of the multimedia file and the frame rate provided for the main object of the multimedia file.

[0078] Combination Figure 5A ,exist Figure 5DIn the example of FIG. 5 , an increased frame rate of 240 FPS is also displayed on the timeline of the multimedia file, along with a unique identifier (such as a graphical representation with a color change of the timeline), compared to the normal frame rate of the multimedia file at 502d. Similarly, the frame rate at which the secondary object is captured, i.e., 120 FPS, is also displayed on the timeline of the multimedia file. However, the frame rate used to capture the primary object in the scene is always greater than the frame rate used to capture the secondary object in the scene, which is indicated by the change in the unique identifiers of the primary and secondary objects.

[0079] Figure 6 is an example illustration of detecting shape events and sound events associated with a balloon and recording a multimedia file in at least one of a recording mode and an effect when no secondary objects are present within a field of view (FOV) of a camera according to an embodiment disclosed herein.

[0080] refer to Figure 6 At 602, the electronic device 100 provides a preview of a scene including a primary object (i.e., a balloon) within the FOV of the camera 110 of the electronic device 100. At step 604, the user identifies the balloon as the primary object in the scene preview. In addition, the electronic device 100 can also automatically identify the primary object in the preview of the scene.

[0081] In step 606, the electronic device 100 suggests a secondary object that is contextually related to the primary object in the scene and is not present in the scene preview displayed in the camera 110. In the case where the electronic device 100 does not suggest a secondary object that the user wants to introduce into the scene, the user can provide a voice command to the electronic device 100 to add the secondary object. In step 608, the user selects the desired secondary object as a bullet. In steps 610 and 612, the electronic device 100 detects a shape event and determines a sound event associated with the primary object due to the secondary object, i.e., when the bullet hits the balloon, the balloon explodes, resulting in a sudden sound and instantaneous change in shape. In addition, the electronic device 100 determines the recording mode and effect that can be applied to capture the primary object and the secondary object, and records the video after applying the recording mode and effect. The explosion of the balloon can be captured in a recording mode such as slow motion, and has an effect such as focusing to clearly capture the change in the shape and sound of the balloon.

[0082] In conventional methods and systems, the user must manually change the recording mode / effect while capturing an event. Therefore, there is a possibility of missing an event / moment important to the user while the user is identifying the recording mode / effect, etc. Unlike conventional methods and systems, the proposed method automatically determines the recording mode / effect to be applied while capturing an event based on a sudden sound change / shape change of a main object, which is of interest to the user.

[0083] Figure 7 is an example illustration of detecting shape events and sound events associated with a bow and recording a multimedia file in at least one of a recording mode and an effect according to embodiments disclosed herein.

[0084] refer to Figure 7 , in step 702, it is considered that the user is previewing an archer using a bow and arrow within the field of view of the camera of the electronic device 100. The electronic device 100 automatically detects multiple objects within the field of view of the camera, namely the archer and the bow. In addition, the user selects the bow as the main object in the scene. In step 704, the electronic device 100 automatically determines the secondary object associated with the main object context and displays the suggested secondary object to the user. The user selects a secondary object selected from a plurality of secondary objects provided by the electronic device 100. If the electronic device 100 does not provide the secondary object selected by the user, the user can add the secondary object by providing a voice command, etc. In addition, in step 706, the electronic device 100 displays the main object and the secondary object on the screen, and also stores the determined main object and the secondary object in the object database. After the selection stage, in step 708, the user starts recording a multimedia file, and the electronic device 100 detects the main object within the field of view of the camera based on the learning. In addition, in step 710, the electronic device 100 stores the main object in the object database.

[0085] In step 712, the electronic device 100 detects shape events and sound events associated with the bow in the scene. Therefore, the electronic device 100 determines the recording mode as slow motion mode based on the sound event and the shape event, and determines the effect as focusing (i.e., zooming in and capturing) to capture the main object. In step 714, the electronic device 100 automatically applies the slow motion mode and the focus effect and records the multimedia file. The slow motion mode applied to the main object while recording is indicated by a unique identifier on the timeline of the multimedia file (i.e., the ascending mode in the timeline). In addition, when the electronic device 100 no longer determines the shape event and the sound event associated with the bow, the electronic device 100 switches to normal mode and continues to record the multimedia file, which is indicated by a straight line in the timeline of the multimedia file. In step 716, when the electronic device 100 receives a frame including a secondary object, a sound verification is performed between the main object and the secondary object. In addition, when a sound event is detected with respect to the secondary object, the electronic device 100 determines the recording mode to be applied to capture the secondary object as slow motion mode, and in step 718, switches to slow motion mode while recording the multimedia file. The slow motion mode applied to the secondary object while recording is indicated by a unique identifier on the timeline of the multimedia file (i.e., the ascending mode on the timeline). However, the height of the ascending mode is smaller when compared to the ascending mode indicating the primary object. This is because the frame rate increase of the primary object is always greater than the frame rate increase of the secondary object. In step 720, when the sound disappears, the electronic device 100 switches to normal mode to record the multimedia file. Once the user has completed the recording of the multimedia file, the multimedia file is stored in the electronic device 100. In addition, when the user plays back the multimedia file, the electronic device 100 automatically displays the unique identifiers corresponding to the recording mode of the primary object and the second recording mode of the secondary object on the timeline of the multimedia file, and plays back the corresponding parts of the multimedia file in their respective recording modes.

[0086] Figure 8 is an example illustration of a method for detecting a shape event associated with a dolphin and recording a multimedia file in at least one of a recording mode and an effect according to embodiments disclosed herein.

[0087] refer to Figure 8, in step 802, it is considered that the user is previewing a scene including a still water scene. In step 804, the electronic device 100 automatically determines an object associated with the scene context, namely a dolphin, and displays the suggested object on the screen of the electronic device 100. The user selects an object selected from a plurality of objects provided by the electronic device 100. If the electronic device 100 does not provide the object selected by the user, the user can add the object by providing a voice command, etc. In addition, the electronic device 100 displays the object on the screen and also stores the determined object in the object database. After the selection stage, in step 806, the user starts recording a multimedia file. In step 808, the electronic device 100 detects a dolphin within the field of view of the camera and starts recording in normal mode. In step 810, the electronic device 100 detects a shape event associated with the dolphin in the scene. Therefore, the electronic device 100 determines the recording mode as a slow motion mode based on the shape event, and determines the effect as focusing (i.e., zooming in and capturing) to capture the dolphin. From steps 810 to 814, the electronic device 100 automatically applies the slow motion mode and the focus effect, and records a multimedia file that captures the entire shape event of the dolphin. The slow motion mode applied to capture the dolphin while recording is indicated by a unique identifier on the timeline of the multimedia file (i.e., the ascending mode on the timeline). In addition, when the electronic device 100 no longer determines the shape event associated with the dolphin, the electronic device 100 switches to the normal mode and continues to record the multimedia file. In addition, when the user plays back the multimedia file including the dolphin, the electronic device 100 automatically displays the unique identifier corresponding to the recording mode of the dolphin on the timeline of the multimedia file and plays back the corresponding part of the multimedia file in the respective recording mode.

[0088] Fig. 9 is an example illustration of detecting a shape event associated with a balloon and recording a multimedia file in at least one of a recording mode and an effect when a secondary object is present in the FOV of a camera of an electronic device according to an embodiment disclosed herein.

[0089] refer to Fig. 9, in step 902, it is considered that the user is previewing a scene including the user's hand approaching a balloon with a pin. The electronic device 100 automatically detects a plurality of objects within the field of view of the camera, namely the balloon and the user's hand. In step 904, the user selects the balloon as at least one object of interest in the scene. In step 906, the electronic device 100 displays the balloon selected by the user as an image on the screen of the electronic device 100, and also stores the determined object in the object database. After the selection stage, in step 908, the user starts recording a multimedia file, and detects a shape event associated with the balloon in step 910. In step 912, the electronic device 100 determines the recording mode as a fast motion mode based on the shape event, and determines the effect as focusing to capture the balloon. From steps 912 to 916, the electronic device 100 automatically applies the fast motion mode and the focus effect, and records a multimedia file capturing the entire shape event of the balloon. The fast motion mode applied to capture the balloon while recording is indicated by a unique identifier on the timeline of the multimedia file (i.e., a descending mode on the timeline). In addition, when the electronic device 100 no longer determines the shape event associated with the balloon, the electronic device 100 switches to the normal mode and continues to record the multimedia file after step 916. In addition, when the user plays back the multimedia file including the balloon, the electronic device 100 automatically displays the unique identifier corresponding to the recording mode of the balloon on the timeline of the multimedia file, and plays back the corresponding part of the multimedia file in the respective recording mode.

[0090] Fig.10 is an example illustration of detecting a shape event associated with a basket in a basketball court and recording a multimedia file in at least one of a recording mode and an effect according to embodiments disclosed herein.

[0091] refer to Fig.10, in step 1002, it is considered that the user is previewing a scene of a basketball court. Since there are multiple objects within the field of view of the camera, the electronic device 100 cannot detect the main object within the scene. Therefore, in step 1004, the user provides a voice input indicating that the main object is a basketball basket, whose shape event must be captured by the electronic device 100. The electronic device 100 displays the basket as an image on the screen of the electronic device 100, and also stores the basket in the object database. After the selection stage, in steps 1006 and 1008, the user starts recording a multimedia file in a normal mode and detects shape events associated with the basket. In step 1010, the electronic device 100 determines the recording mode as a slow motion mode, and determines the effect as focus to capture the shape event associated with the basket. From steps 1010 to 1012, the electronic device 100 automatically applies the slow motion mode and the focus effect, and records a multimedia file that captures the entire shape event of the basket. The slow motion mode applied to capture the balloon while recording is indicated by a unique identifier on the timeline of the multimedia file (i.e., the ascending mode on the timeline). In step 1014, when the electronic device 100 no longer determines the shape event associated with the basket, the electronic device 100 switches to the normal mode and continues to record the multimedia file. In addition, when the user plays back the multimedia file including the basket, the electronic device 100 automatically displays the unique identifier corresponding to the recording mode of the basket on the timeline of the multimedia file and plays back the corresponding part of the multimedia file in the respective recording mode.

[0092] Fig.11 is an example showing selecting a target and automatically recording abnormal actions performed by the target at a higher frame rate according to an embodiment disclosed herein.

[0093] Typically, security applications are highly dependent on tracking various suspicious targets and actions associated with the suspicious targets. Therefore, suspicious targets can be selected based on shape events and sound events, and actions associated with suspicious targets can be automatically recorded at a higher frame rate. In addition, since the frame rate of capturing multimedia files is high, multimedia files including suspicious targets can be analyzed. Consider the situation where a thief enters a store with a gun. In the preview mode of the camera, the user of the electronic device 100 selects an object (i.e., a gun) that needs to be monitored. In step 1102, the electronic device 100 determines that the gun suddenly appears in the scene being captured within the field of view of the camera. In addition, in step 1104, the electronic device 100 determines the shape event associated with the gun and records the multimedia file at an increased frame rate.

[0094] Fig.12 is an example illustration of detecting a shape event associated with a balloon and recording a multimedia file in at least one of a recording mode and an effect when a secondary object is present within the FOV of a camera of a wearable device according to an embodiment disclosed herein.

[0095] Combination Fig. 9 ,refer to Fig.12 , at step 1202, the wearable device 100 with a camera can also be used to preview a scene including multiple objects. In addition, after the selection stage, from steps 1204 to 1208, the user records a multimedia file and detects a shape event associated with the balloon. The wearable device 100 determines a recording mode and effect for capturing the balloon based on the shape event. In addition, the wearable device 100 automatically applies the determined recording mode and effect and records the multimedia file. The recording mode applied to the balloon while recording is indicated by a unique identifier (i.e., a pie chart) on the screen of the wearable device 100.

[0096] Fig.13 is an example illustration of a method for detecting shape events and sound events associated with a vehicle and recording a multimedia file in at least one of a recording mode and an effect according to an embodiment disclosed herein.

[0097] refer to Fig.13 , a dashboard camera of a vehicle is considered. The dashboard camera previews the vehicles on the road, as shown in step 1302. In step 1304, the dashboard camera automatically determines the specific vehicle that is speeding as the main object. In step 1306, the dashboard camera starts recording a multimedia file containing the main object. In step 1308, the electronic device 100 detects a sudden shape event and a sound event associated with the main object caused by the main object (i.e., the speeding vehicle) hitting another vehicle on the road. The dashboard camera starts recording the multimedia file in a slow motion mode with a focus effect to capture the license plate number details, etc. of the main object. In step 1310, the vehicle with the dashboard camera exceeds the main object, and the dashboard camera continues to record the multimedia file containing the vehicle on the road in a normal mode. In addition, when the user plays back the multimedia file including the speeding vehicle, the electronic device 100 automatically displays the unique identifier corresponding to the recording mode of the speeding vehicle on the timeline of the multimedia file, and plays back the corresponding part of the multimedia file in the respective recording mode. Therefore, the proposed method can be used by law enforcement agencies to easily track vehicles violating traffic, because the dashboard camera in the vehicle can be used to capture the license plate details of the traffic violating vehicle.

[0098] Fig.1414 is an example diagram for capturing multiple objects by recording multiple multimedia files using multiple cameras with at least one of recording modes and effects, and generating a multimedia file by merging the multiple multimedia files according to an embodiment disclosed herein. An electronic device 100 having more than one camera can be used to capture a primary object and a secondary object using a first camera and a second camera of the more than one camera, respectively. In step 1402, a user selects a primary object from a view provided by a first camera of the electronic device 100, and selects a secondary object from a view provided by a second camera. In step 1404, the electronic device 100 tracks the primary object and determines at least one of a shape event and a sound event associated with the primary object. In addition, the electronic device 100 uses the first camera. In addition, the electronic device 100 also uses the second camera to track the v object and determines at least one of a shape event and a sound event associated with the secondary object. In addition, as shown in steps 1404 and 1406, while recording the multimedia file, the electronic device 100 determines and applies the recording mode and effect to be applied to the primary object and the secondary object through each camera. Furthermore, in step 1408, the electronic device 100 merges the frames of the primary object recorded using the first camera and the frames of the secondary object recorded using the second camera to generate a single multimedia file including recording modes and effects applied to both the primary object and the secondary object.

[0099] The foregoing description of specific embodiments will reveal the general nature of the embodiments herein so fully that others can easily modify and / or adapt to various applications of this specific embodiment by applying current knowledge without departing from the general concept, and therefore, such adaptations and modifications should and are intended to be understood as being within the meaning and scope of equivalents of the disclosed embodiments. It should be understood that the wording or terminology employed herein is for descriptive purposes and not limiting. Therefore, although the embodiments herein have been described according to preferred embodiments, those skilled in the art will recognize that the embodiments herein can be practiced with modifications within the spirit and scope of the embodiments as described herein.

Claims

1. A method for recording multimedia files using an electronic device, the method comprising: previewing a scene including a plurality of objects in a field of view of a camera of the electronic device; detecting at least one of a shape event and a sound event associated with a primary object among the plurality of objects in the scene due to a secondary object selected by a user of the electronic device, wherein the shape event includes a deformation of the primary object or a change in a state of movement of the primary object, and the secondary object is related to the primary object and does not exist in the preview scene; determining at least one of a recording mode and an effect of the main object based on at least one of the sound event and the shape event; applying at least one of the recording mode and the effect; recording the multimedia file including the primary object and the secondary object in the at least one of the recording mode and the effect; and The recorded multimedia file is stored.

2. The method according to claim 1, wherein: Detecting a sound event associated with the primary object includes: determining a sound pressure level SPL of a sound associated with the primary object; determining whether the SPL of the sound associated with the primary object satisfies a sound change criterion; and The sound event associated with the primary object is detected when the SPL of the sound associated with the primary object satisfies the sound change criterion.

3. The method according to claim 1, wherein: Detecting a shape event associated with the primary object includes: obtaining a plurality of frames associated with the primary object; determining a difference of at least one region of the primary object in each of the frames based on a distance from a centroid of the primary object to each edge; determining a rate of change of at least one region of the main object in each of the frames based on a distance from a centroid of the main object to each of the edges; determining whether a rate of change of the at least one region of the main object in each of the frames satisfies a shape criterion; and When the rate of change satisfies the shape criterion, the shape event associated with the primary object is detected.

4. The method according to claim 1, wherein: Determining at least one of a recording mode and an effect of the main object based on at least one of the sound event and the shape event includes: determining a frame rate based on at least one of the shape event and the sound event associated with the primary object; and At least one of the recording mode and the effect of the main subject is determined based on the frame rate.

5. The method according to claim 1, wherein: The secondary object is determined dynamically by: providing at least one preview frame of the scene including the primary object to the machine learning model; Determining the context of the primary object according to a CNN-based model; and The secondary object associated with the primary object is determined based on the context.

6. A method for recording multimedia files using an electronic device, the method comprising: recording a first portion of a scene in at least one of a first recording mode and a first effect, wherein while recording the first portion of the scene, a first identifier corresponding to the first recording mode is displayed on a timeline of the multimedia file; While recording the scene, detecting at least one of a shape event and a sound event associated with a primary object among a plurality of objects in the scene due to a secondary object, wherein the shape event includes a deformation of the primary object or a change in a state of movement of the primary object, and the secondary object is related to the primary object and does not exist in the preview scene; determining at least one of a second recording mode and a second effect based on the shape event and the sound event; recording a second portion of the scene including the primary object and the secondary object in at least one of the second recording mode and the second effect, wherein while the second portion of the scene is being recorded, a second identifier corresponding to the second recording mode is displayed on a timeline of the multimedia file; generating a multimedia file including the first portion of the scene and the second portion of the scene; and storing the multimedia file in the electronic device, Wherein, the secondary object is selected by a user of the electronic device.

7. The method according to claim 6, further comprising: Playing back the multimedia file; displaying the first identifier and the second identifier on a timeline of the multimedia file; as well as The first portion is played back in the first recording mode and the second portion is played back in the second recording mode.

8. The method according to claim 6, wherein: Recording the first portion of the scene in at least one of the first recording mode and the first effect includes: detecting at least one of the shape event and the sound event associated with the primary object among the plurality of objects in the scene; determining at least one of a first recording mode and a first effect of the main object based on at least one of the sound event and the shape event; applying at least one of the first recording mode and the first effect; and The first portion of the scene is recorded in at least one of the first recording mode and the first effect.

9. The method according to claim 6, wherein: Determining at least one of the second recording mode and the second effect based on the shape event and the sound event includes: determining a frame rate based on at least one of the shape event and the sound event associated with the primary object; and At least one of the second recording mode and the second effect of the main object is determined based on the frame rate.

10. The method according to claim 6, wherein: Detecting a sound event associated with the primary object includes: determining a sound pressure level SPL of a sound associated with the primary object; determining whether the SPL of the sound associated with the primary object satisfies a sound change criterion; and The sound event associated with the primary object is detected when the SPL of the sound associated with the primary object satisfies the sound change criterion.

11. An electronic device for recording multimedia files, the electronic device comprising: Memory; a processor coupled to the memory; as well as a recording mode management engine, the recording mode management engine being coupled to the memory and the processor, wherein the recording mode management engine is configured to: previewing a scene including a plurality of objects in a field of view of a camera of the electronic device; detecting at least one of a shape event and a sound event associated with a primary object among the plurality of objects in the scene due to a secondary object selected by a user of the electronic device, wherein the shape event includes a deformation of the primary object or a change in a state of movement of the primary object, and the secondary object is related to the primary object and does not exist in the preview scene; determining at least one of a recording mode and an effect of the main object based on at least one of the sound event and the shape event; applying at least one of the recording mode and the effect; recording the multimedia file including the primary object and the secondary object in the at least one of the recording mode and the effect; and The recorded multimedia files are stored.

12. The electronic device according to claim 11, wherein: The recording mode management engine is configured to: determining a sound pressure level SPL of a sound associated with the primary object; determining whether the SPL of the sound associated with the primary object satisfies a sound change criterion; and The sound event associated with the primary object is detected when the SPL of the sound associated with the primary object satisfies the sound change criterion.

13. The electronic device according to claim 11, wherein: The recording mode management engine is configured to: determining a frame rate based on at least one of the shape event and the sound event associated with the primary object; as well as At least one of the recording mode and the effect of the main subject is determined based on the frame rate.

14. An electronic device for recording multimedia files, the electronic device comprising: Memory; a processor coupled to the memory; as well as a recording mode management engine, the recording mode management engine being coupled to the memory and the processor, wherein the recording mode management engine is configured to: recording a first portion of a scene in at least one of a first recording mode and a first effect, wherein while recording the first portion of the scene, a first identifier corresponding to the first recording mode is displayed on a timeline of the multimedia file; While recording the scene, detecting at least one of a shape event and a sound event associated with a primary object among a plurality of objects in the scene due to a secondary object, wherein the shape event includes a deformation of the primary object or a change in a state of movement of the primary object, and the secondary object is related to the primary object and does not exist in the preview scene; determining at least one of a second recording mode and a second effect based on the shape event and the sound event; recording a second portion of the scene including the primary object and the secondary object in at least one of the second recording mode and the second effect, wherein a second identifier corresponding to the second recording mode is displayed on a timeline of the multimedia file while the second portion of the scene is being recorded; generating a multimedia file including the first portion of the scene and the second portion of the scene; and storing the multimedia file, Wherein, the secondary object is selected by a user of the electronic device.

15. The electronic device according to claim 14, wherein: The recording mode management engine is configured to: determining a sound pressure level SPL of a sound associated with the primary object; determining whether the SPL of the sound associated with the primary object satisfies a sound change criterion; and The sound event associated with the primary object is detected when the SPL of the sound associated with the primary object satisfies the sound change criterion.

Images