Behaviour recognition

Abstract

The invention relates to a method for use in assessing eye contact of a subject participant in a social interaction scenario between participants. The method involves using camera apparatus to capture video images of the subject participant as a recorded image stream, processing the captured video images to provide data representing spatial and temporal features of the recorded image stream, and using a computer processing module to apply an action recognition algorithm to the data to predict occurrences of eye contact of the subject participant. The invention has particular application in evaluation of children, to assess development of social skills in a simulated, real-life interaction scenario such as a free play task guided by a facilitator in a way that prompts eye contact by the subject participant child.

Description

BEHAVIOUR RECOGNITIONTECHNICAL FIELD

[0001] The present disclosure generally relates to behaviour recognition, and in particular methods for eye contact determination in human subjects by analysing video capture of the interactions of human subjects.BACKGROUND

[0002] Eye contact is an important non-verbal communication behaviour in social interaction that assists with understanding other people’s social attention, emotional changes, relationship intimacy, and trust. For example, studies have found that attention and eye contact are increased during interactions when people feel emotionally aroused about the other person or when they have expectations for a response from the other.

[0003] In addition to affecting relationship quality, other research has pointed to the role eye gaze plays in clinical disorders and mental well-being. Eye contact is also believed to contribute to early social development as there is evidence that reduced eye contact early in life is often associated with brain and cognitive divergence.

[0004] When an observation is made that a care recipient, such as a child, may have an undiagnosed disorder such as autism spectrum disorder (ASD), it is important to obtain a rapid and reliable diagnosis. Currently, tests for ASD diagnosis are lengthy and cumbersome, and clinical facilities and professionals configured for administering and assessing such tests are too few, meaning that such conditions are often diagnosed too late for effective treatments to be initiated at a stage where intervention is most likely to have a positive impact.

[0005] Further, by way of background context, the quality of social interactions between children and their caregivers is often studied in free play environments. Free play allows observation of a child’s natural interactions with other people and objects in their surroundings in an environment that does not restrict their movements. It is also used in the assessment of clinical conditions as a standard social interaction task. Free play in early childhood contributes to the development of individual adaptivity, social skills, mental health, and increased creative thinking through self-directed activities that enhance cognitive and brain development.

[0006] During such more naturalistic play tasks, measuring eye contact in social interactions traditionally involved an observer using video recordings of play sessions to identify and countthe frequency and duration of eye contacts between the human subjects. This is time-consuming, prone to error and misclassification of events, which make such measures resource-intensive and impractical for everyday clinical use.

[0007] An automated method currently available relies on eye trackers fixed to a computer screen to record facial and eye movements during an experimental session. Some other automated methods rely on robotic-assisted treatment mechanisms that interact socially with the participants through remote controlling of the robot by an observer. However, these methods do not directly analyse real human social interaction and require observing the front of the participants’ face and particularly the participant’s eyes in order to detect relevant social biomarkers. Such solutions do not therefore lend themselves to use in free play scenarios and add a significant artificial aspect to the environment. Such approaches also have additional requirements for calibration that can sometimes make it difficult to collect data from different samples (e.g., where young children are involved) and different sites (e.g., different room environments).

[0008] Other automated methods rely on wearable technology. For example, an egocentric camera may be embedded within wearable glasses to record participants’ gaze direction. In such methods, the definition of an event of eye contact between two participants is the act of both participants looking at the eye region of one another - both participants wearing the functional glasses and each glass worn by a participant recording actions of the other participant. The limitation of such wearable technology is that the glasses move with the wearer, which imposes restriction on a free play task. In addition, it adds additional devices to the face that is the focus of social interactions, which potentially affects facial view and therefore social interaction. It also may be difficult to use such devices when the person wearing them may have difficulty with the wearable technology (e.g., young children and people with sensory sensitivities of glasses).

[0009] An example of a prior art automated solution is US Patent No. 10,966,605, concerning a method of deriving features of a subject from captured eye movement signals (e.g., from headmounted eye-trackers), involving comparing with biometric templates created from the records of multiple healthy people. As a further example, US Patent No. 10,405,786 discloses a method of evaluation of autism in an individual, conducted in the home environment, involving data collection by a wearable data collection device (such as electronic eyewear, or a visor, hat or headband) donned by the individual able to monitor eye contact, verbalisations and otherinteractions of the individual with a caregiver, such data analysed by the system to recognise patterns that predict clinical indications of autism.

[0010] In common with other known approaches, the two examples above suffer from the significant disadvantage of the requirement of special equipment to be worn by one or more subject(s), thus interfering with a natural free-play environment and potentially influencing the subjects’ behaviour during the session.

[0011] The present disclosure is generally concerned with determination of eye contact between human subjects based on video footages of interactions between the human subjects.

[0012] Reference to any prior art in the specification is not an acknowledgement or suggestion that this prior art forms part of the common general knowledge in any jurisdiction or that this prior art could reasonably be expected to be combined with any other piece of prior art by a skilled person in the art.SUMMARY

[0013] In a first aspect, the invention provides a method for use in assessing eye contact of a subject participant in a social interaction scenario between participants, the method involving: using camera apparatus to capture video images of said subject participant as a recorded image stream; processing said captured video images to provide data representing spatial and temporal features of the recorded image stream; and using a computer processing module to apply an action recognition algorithm to said data to predict occurrences of eye contact of said subject participant.

[0014] As will be appreciated, in accordance with the particular solution provided by the invention, the prediction of occurrences of eye contact of said subject participant is made irrespective of whether the eyes of said subject participant are visible in the captured video images.

[0015] In one embodiment, the subject participant is a child and a further participant is a facilitator, the social interaction scenario being a free play task adapted to assess social skills of the child interacting with the adult facilitator.

[0016] This approach allows evaluation of the subject participant child in a simulated, real-life interaction scenario, which serves to prompt eye contact from the subject participant child and is generally controllable by the facilitator further participant.

[0017] Preferably, the method includes using multiple camera apparatuses to capture video images of said subject participant from multiple angles.

[0018] The multiple camera apparatus are preferably positioned to collectively produce a panoramic view of the activities carried out by and the social interactions between the participants.

[0019] As will be understood, the computer processing module is used to analyse the recorded image stream and evaluate participant actions predictive of eye contact.

[0020] The video images may include or be associated with metadata including one or more of timestamp data and camera identifier data.

[0021] Preferably, the method involves determining prediction scores based on temporal properties of said recorded image stream.

[0022] In one form, the method includes dividing the recorded image stream into a plurality of segments of prescribed length and predicting occurrences of eye contact of said subject participant in each segment.

[0023] In a preferred form, the recorded image stream is decoded into two data sets:(a) a spatial frame data set, representing 2D spatial information from an array of pixels in a frame or a sequence of frames and(b) an optical flow data set, being a data set associated with flow vector patterns representing apparent motion of the content of an image between frames in a sequence of frames, the method including: analysing the spatial frame data set to obtain one or more first prediction scores associated with occurrence of eye contact of said subject participant, analysing the optical flow data set to obtain one or more second prediction scores associated with occurrence of eye contact of said subject participant; and combining said one or more first prediction scores and said one or more second prediction scores to obtain one or more aggregate prediction scores.

[0024] The optical flow data set may comprise a grayscale optical flow image stack.

[0025] When the method includes using multiple camera apparatuses to capture video images of said subject participant from multiple angles, respective spatial frame data sets and optical flow data sets are obtained for the recorded image stream from each camera apparatus and analysed independently to obtain consensus one or more first prediction scores and consensus one or more second prediction scores, such consensus scores being used to determine said one or more aggregate predictions scores.

[0026] In this aspect, as will be understood, where the method includes dividing the recorded image stream into a plurality of segments of prescribed length and predicting occurrences of eye contact of said subject participant in each segment, one or more aggregate predictions scores may be calculated for each segment.

[0027] The method may include analysing predicted occurrences of eye contact of said subject participant in multiple segments from the recorded image stream, in order to provide information useful for assessment one or more behavioural issues concerning the subject participant.

[0028] In a further aspect, the invention provides a method for use in training an action recognition algorithm to predict occurrences of eye contact of a subject participant in a social interaction scenario between participants, the method involving: processing multiple recorded image streams comprising captured video images of a number of subject participants in social interaction scenarios between participants, to provide data representing spatial and temporal features of each of the multiple recorded image streams; and classifying aspects of each of the multiple recorded image streams as representative or not of occurrences of eye contact of the subject participant in the respective image stream, for use in an eye contact prediction method.

[0029] Preferably, each of the multiple recorded image streams is decoded into two data sets:(a) a spatial frame data set, representing 2D spatial information from an array of pixels in a frame or a sequence of frames and(b) an optical flow data set, being a data set associated with flow vector patterns representing apparent motion of the content of an image between frames in a sequence of frames, the method including classifying each spatial frame data set as representative or not of occurrence of eye contact of the subject participant in the image stream, classifying each opticalflow data set as representative or not of occurrence of eye contact of the subject participant in the image stream, both said classifications for use in the eye contact prediction method.

[0030] Various methods, apparatuses, and systems are configured to provide techniques for automatically detecting eye contacts between human subjects based on single- view or multi- view video footages of an event of interest (e.g., a free play session). The inventors have identified deficiencies and problems associated with existing methods, apparatuses, and systems for automatically detecting eye contact between human subjects based on video footages of an event of interest. Through applied effort, ingenuity, and innovation, these identified deficiencies and problems have been solved by developing solutions that are in accordance with the embodiments of the present invention, many examples of which are described in detail herein.BRIEF DESCRIPTION OF THE DRAWINGS

[0031] By way of non-limiting example, the accompanying drawings illustrate one or more embodiments of the present invention.

[0032] Fig. 1 is a diagram depicting a networked environment in which various features of the present disclosure may be implemented.

[0033] Fig. 2 is a diagram depicting a Multi-view Social Behavior Recognition (MSBR) System according to aspects of the present disclosure.

[0034] Fig. 3 is a block diagram of a computer processing system configured to perform various features of the present disclosure.

[0035] Fig. 4 is a schematic illustrating an environment in which multi-view video footages of an event of interest are recorded.

[0036] Fig. 5 is a flowchart illustrating an example method for detecting eye contact in multiview video footages according to some aspects of the present disclosure.

[0037] Fig. 6 is a flowchart illustrating an example method for decoding multi- view video footages according to some aspects of the present disclosure.

[0038] Fig. 7 is a flowchart illustrating an example method for computing single-view prediction scores according to some aspects of the present disclosure.

[0039] Fig. 8 is a flowchart illustrating an example method for aggregating prediction scores used for determining eye contact according to some aspects of the present disclosure.

[0040] Fig. 9 is a diagram depicting the flow of data and the processes according to some aspects of the present disclosure.

[0041] While the invention described herein is amenable to various modifications and alternative forms, specific embodiments are shown by way of example in the drawings and detailed description. It should be understood, however, that the drawings and detailed description are not intended to limit the invention to the particular form disclosed. The intention is to cover all modifications, equivalents, and alternatives falling within the spirit or scope of the present invention as claimed.DETAILED DESCRIPTION

[0042] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the claimed invention. It will be apparent, however, that the claimed invention may be realised in alternative ways, without the specific details set out below.

[0043] The present disclosure is generally concerned with recognition of eye contact between human subjects based on video footages of interactions between the human subjects during an event of interest. There may only be one video footage representing one view of an event of interest. Alternatively, there may be multi-view video footages that represent an event of interest from various views (e.g., multiple camera devices installed on walls of the room where interactions occur between human participants) supported by additional data (e.g., metadata such as timestamp data and camera view identifier data).

[0044] Recognition of eye contact events is a form of indirect determination from the information captured in one or more video footages (e.g., body postures, direction and duration of gaze) without necessarily requiring the participants’ eyes being visible in the video footages.

[0045] An example event of interest may be a free play task for assessing social skills of a child interacting with an adult assessor in a simulated real-life interaction scenario. The free play task allows the participants to play in a natural social interaction environment where there are no restrictions on their movement in the trial room. In particular, the present invention does not require the participants to wear or carry any particular equipment, thus ensuring the scenario is as wholly natural as possible.

[0046] The examples and scenarios described herein concern human participants; however the techniques of the invention can equally be used in recognising eye contact events involving animal participants, such as domestic animals, farm animals, etc.

[0047] Each video footage (or each segment of video footage) included in the set of multi- view video footages is decoded into RGB and optical flow images representing spatial and temporal features of the video footage at image level.

[0048] Each RGB image represents one video frame consisting of a grid of pixels, with each pixel containing colour information, the colour of each pixel determined by a combination of three colour channels, red, green and blue. Each RGB image therefore contains 2D spatial information for that frame.

[0049] Decoding a video into optical flow images involves determining the apparent motion between consecutive frames in the video, by analysis of the pixel intensity patterns. In other words, optical flow is a flow vector pattern representing apparent motion of the contents (objects, surfaces, edges) of an image from one frame to the next, caused (in the case of a fixed camera) by movement of the object(s) in the frame. For example, consider two consecutive frames from a video footage showing an object moving from left to right. In the optical flow image, pixels representing the moving object will be associated with motion vectors pointing towards the right, the length and direction of motion vectors indicating the speed and direction of the object's motion for that pixel. Optical flow images associated with a video footage thus provide temporal information concerning the developing scene in the video footage.

[0050] In one embodiment, RGB and optical flow images are processed using an action recognition algorithm for predicting actions occurring during an event of interest (e.g., whether an eye contact occurs during a segment).

[0051] In one embodiment, if there is more than one segment per video footage, prediction scores obtained through action recognition algorithm may be aggregated to obtain footage-level prediction scores. Footage-level prediction scores are further aggregated to obtain multi-view prediction scores. Multi-view prediction scores are then used to determine if eye contact is present in an event of interest. This method allows simultaneous recognition of special-temporal features of eye contact behaviour.

[0052] In some embodiments, each video footage included in multi- view video footages is divided into an equal number of segments with an equal length (e.g., segments of a few frames). Each video segment is decoded into RGB and optical flow images and then processed using action recognition algorithm for predicting actions occurring during an event of interest. In some embodiments, RGB and optical flow consensus scores are separately computed for each videosegment. In some embodiments, fusion scores are computed based on segment-level RGB and optical flow consensus scores. In some embodiments, an average of fusion scores associated with video segments of multi- view video footages is used to determine if eye contact is present in an event of interest.

[0053] It will be appreciated that the length of video segments associated with each video footage may be very short. For example, a video footage may be divided into a number of segment such that each segment only contains a small number of video frames corresponding to a fraction of a second or a few seconds of the video footage.

[0054] Analysis of video segments by way of the methods described herein results in a series of eye-detection results for a series of segments, which series can further be analysed for diagnostic purposes (e.g., to diagnose autism in a child participant).System overview

[0055] Fig. 1 depicts an exemplary environment 100 for recording multi-view video footages of an event of interest and detecting potential eye contacts between human subjects.Environment 100 includes one or more camera devices 110A, 110B, 110C and 110D (collectively referred to as camera devices 110), a recording system 120, and a Multi-view Social Behaviour Recognition (MSBR) System 130.

[0056] In general, camera devices 110 are suitable for recording social interactions between the human subjects during an event of interest. These may be high-definition (HD) camera recorders and may also be capable of recording audio through integrated microphones.

[0057] In one embodiment, camera devices 110 may optionally include two or more camera devices (e.g., 110A-110D) that provide video footages of an event of interest from a number of view angles. In other embodiments, there may be only one camera device (e.g. 110A) that represents an event of interest.

[0058] Recording system 120 is a system entity that hosts a recording application configured to receive and manage raw video data from camera devices 110. System 120 may include one or more servers (e.g., 122) for hosting the recording application and one or more storage devices (e.g., database 124) for storing application specific data (e.g., video data and metadata).

[0059] Multi-view Social Behaviour Recognition System (MSBR) 130 is communicatively coupled with recording system 120. In particular, it receives multi-view video footages togetherwith other associated data from system 120, analyses this content to determine whether the content indicates a potential eye contact between the human subjects present in the video footages. In order to do so, MSBR system 130 includes a plurality of sub-systems or modules.

[0060] As illustrated in FIG. 1, communications between the various systems 110-130 are via the communication network 140. The communication network 140 is depicted as a single network in Fig. 1 for ease of depiction. However, in actual implementation, the various systems illustrated in FIG. 1 may communicate with each other over different communication networks. For example, Multi-view Social Behavior Recognition (MSBR) System 130 may communicate with recording system 120 through a local area network (LAN), whereas it may communicate with camera devices 110 via a public network (e.g., the Internet).

[0061] While single server architecture has been described herein, it will be appreciated that recording system 120 and multi-view social behavior recognition system 130 can be implemented using alternative architectures. For example, in certain cases a clustered architecture may be used where multiple server computing instances (or nodes) are instantiated to meet system demand. Conversely, in the case of small enterprises with relatively simple requirements the recording system 120 and multi-view social behavior recognition system 130 may be a stand-alone implementations (i.e., single server computers) directly accessed / used by the end user.

[0062] It will be appreciated that although the environment 100 shows four camera devices 110 and one recording system 120, this is only one example of a possible implementation. In other implementations fewer or more camera devices 110 may be interfaced with MSBR system 130. Similarly, in those or other implementations, more than one recording system 120 may be interfaced with MSBR system 130. Further, each of those recording systems may monitor one or more camera devices without departing from the scope of the present disclosure.

[0063] Further, although MSBR system 130 is depicted as an independent entity in communication with recording system 120, this need not be the case in all implementations. In some cases, MSBR system 130 may be part of recording system 120 and may be implemented by recording system server 122.

[0064] Turning to Fig. 2, a block diagram 200 is shown that illustrates the sub-systems or modules of MSBR system 130. In particular, as seen in Fig. 2, MSBR system 130 includes a pre-processing module 202, RGB prediction module 204, optical flow prediction module 206, prediction aggregator module 208, and eye contact detection module 210.

[0065] As noted earlier, MSBR system 130 receives content including multi-view video footages and the associated data from recording system 120 to determine whether such content indicates potential eye contacts between human subjects.

[0066] Pre-processing module 202 receives multi-view video footages and the associated data from system 120. Multi-view video footages include video footages that represent an event of interest from at least one view. Availability of more views of an event of interest from various angles (e.g., through camera devices 110) provides more data that MSBR system 130 can analyse and aggregate to improve accuracy of recognising eye contact events.

[0067] Pre-processing module 202 may carry out a number of pre-processing actions on the videos to prepare them for processing by other modules of MSBR system 130. For example, module 202 may ensure that multi- view video footages are valid video files and in an appropriate video format / standard. Module 202 may also include an optional step of dividing each video footage into smaller segments. As will be described later, this optional step may be advantages to optimise the required processing for eye contact detection.

[0068] RGB prediction module 204 receives multi- view video footages from pre-processing module 202 and determines prediction scores based on spatial properties of video footages per video footage (or per segment of each video footage). Details of the steps involved in computing the RGB prediction scores are later described in relation to methods 504 and 506.

[0069] Optical flow prediction module 206 receives multi- view video footages from preprocessing module 202 and determines prediction scores based on temporal properties of video footages. The prediction scores computed by module 206 are at video footage level (or at video segment level if each footage is divided into segments). Details of the steps involved in computing the optical flow prediction scores are later described in relation to methods 504 and 506.

[0070] As described below in relation to method steps 504 and 506, RGB prediction module 204 and optical flow prediction module 206 use an action prediction algorithm to compute prediction scores associated with video footages. Modules 204 and 206 receive multi-view video footages, decode them into RGB and optical flow images, and input these images into neuralnetworks (e.g. convolutional neural networks (CNNs)) in order to compute the desired prediction scores. Details of the technical steps involved in training and using a CNN are described below.

[0071] Prediction aggregator module 208 receives and aggregates spatial and temporal prediction scores determined by modules 204 and 206. This approach enables system 130 to combine the learnings from spatial and temporal analysis. Details of the steps involved in aggregating prediction scores is described later in relation to method 508.

[0072] Eye contact detection module 210 receives and processes aggregated prediction scores from module 208 to detect the presence of eye contact between the human subjects participating in the event of interest. Details of the steps involved to detect eye contacts is described later in relation to method 508.Hardware overview

[0073] The embodiments and features described herein are implemented by one or more special-purpose computing systems or devices. For example, in environment 100 each of camera devices 110 is or includes a type of computing device and each of recording system 120 and MSBR system 130 is or includes a type of computing system.

[0074] A special-purpose computing system may be hard-wired to perform the relevant operations. Alternatively, a special-purpose computing system may include digital electronic devices such as one or more application-specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs) that are persistently programmed to perform the relevant operations. Further, alternatively, a special-purpose computing system may include one or more general-purpose hardware processors programmed to perform the relevant operations pursuant to program instructions stored in firmware, memory, other storage, or a combination.

[0075] A special-purpose computing system may also combine custom hard-wired logic, ASICs, or FPGAs with custom programming to accomplish the relevant operations described herein. This may, for example, assist with privacy preservation at the edge. A special-purpose computing system may be a desktop computer system, a portable computer system, a handheld device, a networking device or any other device that incorporates hard-wired and / or program logic to implement relevant operations.

[0076] Fig. 3 provides a block diagram of a computer processing system 300 configurable to implement embodiments and / or features described herein. System 300 is a general purpose computer processing system. It will be appreciated that Fig. 3 does not illustrate all functional orphysical components of a computer processing system. For example, no power supply or power supply interface has been depicted, however system 300 will either carry a power supply or be configured for connection to a power supply (or both). It will also be appreciated that the particular type of computer processing system will determine the appropriate hardware and architecture, and alternative computer processing systems suitable for implementing features of the present disclosure may have additional, alternative, or fewer components than those depicted.

[0077] Computer processing system 300 includes at least one processing unit 302, for example a general or central processing unit, a graphics processing unit, or an alternative computational device). Computer processing system 300 may include a plurality of computer processing units. In some instances, where a computer processing system 300 is described as performing an operation or function, all processing required to perform that operation or function will be performed by processing unit 302. In other instances, processing required to perform that operation or function may also be performed by remote processing devices accessible to and useable by (either in a shared or dedicated manner) system 300.

[0078] Through a communications bus 304, processing unit 302 is in data communication with one or more computer readable storage devices which store instructions and / or data for controlling operation of the processing system 300. In this example, system 300 includes a system memory 306 (e.g. a BIOS), volatile memory 308 (e.g. random access memory such as one or more DRAMs), and non-volatile (or non-transitory) memory 310 (e.g. one or more hard disks, solid state drives, or other non-transitory computer readable media). Such memory devices may also be referred to as computer readable storage media (or a computer readable medium).

[0079] System 300 also includes one or more interfaces, indicated generally by 312, via which system 300 interfaces with various devices and / or networks. Generally speaking, other devices may be integral with system 300, or may be separate. Where a device is separate from system 300, connection between the device and system 300 may be via wired or wireless hardware and communication protocols, and may be a direct or an indirect (e.g. networked) connection.

[0080] Wired connection with other devices / networks may be by any appropriate standard or proprietary hardware and connectivity protocols, for example Universal Serial Bus (USB), eSATA, Thunderbolt, Ethernet, HDMI, and / or any other wired connection hardware / connectivity protocols.

[0081] Wireless connection with other devices / networks may similarly be by any appropriate standard or proprietary hardware and communications protocols, for example infrared, BlueTooth, Wi-Fi, near field communications (NFC), Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), long term evolution (LTE), code division multiple access (CDMA - and / or variants thereof), and / or any other wireless hardware / connectivity protocol.

[0082] Generally speaking, and depending on the particular system in question, devices to which system 300 connects - whether by wired or wireless means - include one or more input / output devices (indicated generally by input / output device interface 314). Input devices are used to input data into system 300 for processing by the processing unit 302. Output devices allow data to be output by system 300. Example input / output devices are described below, however, it will be appreciated that not all computer processing systems will include all mentioned devices, and that additional and alternative devices to those mentioned may well be used.

[0083] For example, system 300 may include or connect to one or more input devices by which information / data is input into (received by) system 300. Such input devices may include keyboards, mice, trackpads (and / or other touch / contact sensing devices, including touch screen displays), microphones, accelerometers, proximity sensors, GPS devices, touch sensors, and / or other input devices. System 300 may also include or connect to one or more output devices controlled by system 300 to output information. Such output devices may include devices such as displays (e.g. cathode ray tube displays, liquid crystal displays, light emitting diode displays, plasma displays, touch screen displays), speakers, vibration applications, light emitting diodes / other lights, and other output devices. System 300 may also include or connect to devices which may act as both input and output devices, for example memory devices / computer readable media (e.g. hard drives, solid state drives, disk drives, compact flash cards, SD cards, and other memory / computer readable media devices) which system 300 can read data from and / or write data to, and touch screen displays which can both display (output) data and receive touch signals (input).

[0084] System 300 also includes one or more communications interfaces 316 for communication with a network, such as network 140 of environment 100. Via a communications interface 316, system 300 can communicate data to and receive data from networked devices, which may themselves be other computer processing systems.

[0085] System 300 may be any suitable computer processing system, for example, a server computer system, a desktop computer, a laptop computer, a netbook computer, a tablet computing device, a mobile / smart phone, a personal digital assistant, or an alternative computer processing system.

[0086] System 300 stores or has access to computer applications (also referred to as software or programs), i.e. computer readable instructions and data which, when executed by the processing unit 302, configure system 300 to receive, process, and output data. Instructions and data can be stored on non-transitory computer readable media accessible to system 300. For example, instructions and data may be stored on non-transitory memory 310. Instructions and data may be transmitted to / received by system 300 via a data signal in a transmission channel enabled (for example) by a wired or wireless network connection over interface such as 312.

[0087] Applications accessible to system 300 will typically include an operating system application such as Microsoft Windows™, Apple macOS™, Apple iOS™, Android™, Unix™, or Linux™.

[0088] System 300 also stores or has access to applications which, when executed by the processing unit 302, configure system 300 to perform various computer-implemented processing operations described herein. For example, and referring to networked environment 100 of Fig. 1 above, recording system 120 includes a server application 122 which configure the recording system 120 to receive video streaming data from camera devices 110 and store such data in database 124, and Multi-view Social Behaviour Recognition (MSBR) System 130 includes a number of subsystems which are executed by one or more processing units 302 which configure the computer processing system(s) to perform the described behaviour recognition system operations.

[0089] In some cases part or all of a given computer-implemented method will be performed by a single computer processing system 300, while in other cases processing may be performed by multiple computer processing systems, local and / or remote, in data communication with each other.Data collection and environment

[0090] Fig. 4 is an example environment 400 where recording system 120 may record multiview video footages of a free play task between human subjects such as a child participant and an adult assessor.

[0091] Environment 400 includes camera devices 110 positioned in different example locations to record an event of interest (e.g., the free play task) from various and complimentary views. By way of example, camera devices 110 may be positioned such that they collectively produce a panoramic view of the activities carried out by and the social interactions between the participants.

[0092] Camera devices 110 may be set up such that, for example, device 110A captures the assessor’s front view, device HOB captures the child’s front view, device HOC captures back view of the child and side view of the assessor, and 110D captures side view of the child and back view of the assessor.

[0093] Recording system 120 (not shown in Fig. 4) receives raw video data from camera devices 110 through network 140. Recording system server 122 processes the raw video data received from camera devices to produce the required complimentary metadata for each video. Such metadata may for example include timestamp data, information related to the participants, and information about the camera device that recorded each video. Server 122 stores the raw video data as multi-view video footages in recording system database 124.Behaviour recognition process

[0094] This section describes a computer- implemented method for eye contact recognition through analysing multi- view video footage associated with an event of interest involving interactions of two or more human participants.

[0095] Fig. 5 is a flowchart illustrating an example method 500 performed by MSBR system 130 to detect eye contact based on multi-view video footages.

[0096] Method 500 commences at step 502 where multi-view video footages are retrieved into volatile memory (e.g. memory 308 of system 300). Such video footages may be retrieved from a local storage (e.g., non-transient memory 310 of system 300) or a remote location accessible through a network connection (e.g., database 124 of recording system 120 accessible through network 140).

[0097] Multi- view video footages consist of at least one video footage per camera (view) representing a distinct view of an event of interest. An event of interest may be captured by one or more camera devices.

[0098] If multi- view video footages include more than one video footage for any of the camera devices, all the video footages associated with that camera device are merged into a single videofootage at step 502. This merging process is intended to ensure there will be one video footage per camera included in multi-view video footages.

[0099] Metadata associated with video footages includes data such as timestamp data, which at least includes a start time and an end time, and data indicating the camera device associated with a footage. To ensure multi-view video footages represent the same event for the same period of time, the start time and end time of all video footages are matched. For example, at step 502 MSBR system 130 modifies all video footages such that none of the footages contains a period of time that does not exist in all other video footages.

[0100] After performing the preparation tasks at step 502, multi-view video footages will have identical start time and end time. All such footages relate to different views recorded of the same event. For example, multi-view video footages may include the following video footages corresponding to each camera device shown in Figs 1 and 4:Video footage Camera Start time End time

[0101] In some embodiments, video footage preparation at step 502 may involve an additional step whereby each video footage (e.g. V1-V4) is further divided into smaller segments. This additional step may provide advantage where multi-view video footages are longer than a certain threshold resulting in excessive processing overhead in the following steps of process 500. In the interest of simplicity in describing the following steps of process 500, the additional step of dividing video footages into segments is described later.

[0102] At step 504, each video footage in the set of multi-view video footages is decoded by MSBR system 130. The process of decoding a video footage involves creating RGB images and optical flow images for the frames in each video footage. This process is described below in relation to Fig. 6.

[0103] At step 506, MSBR system 130 computes single-view prediction scores. This process is described below in relation to Fig. 7.

[0104] At step 508, system 130 aggregates prediction scores and determines the results. This process is described below in relation to Fig. 8.

[0105] Turning to Fig. 6, a flowchart illustrating an example method 504 is described that is performed by MSBR system 130 in order to decode multi-view video footages prepared at step 502.

[0106] The method 504 commences at step 602 where MSBR system 130 selects one video footage representing one view of the event of interest. By way of example, at step 602 MSBR system 130 may select video footage VI as shown in the example above described in relation to step 502.

[0107] At step 604, the selected video footage is decoded into RGB frames, which enables each frame to be represented by its true colours. The resulting RGB images represent spatial domain data associated with each segment and are stored in memory (e.g., memory 310) for further processing by MSBR system 130.

[0108] At steps 606 and 608, the selected video footage segment is decoded into optical flow images for x and y axis, which present the motion pattern between two consecutive frames in x and y axis directions. The resulting motion flow images represent temporal domain data associate with each segment and are stored in memory (e.g., memory 310) for further processing by MSBR system 130.

[0109] Decoding of video segments into RGB frames and optical flow images may be carried out using various tools such as, e.g., DenseFlow python library. As will be understood, a single tool may be used to derive and generate the optical flow images for x and y axes, in which case steps 606 and 608 may be represented as a single, combined step.

[0110] At step 610, system 130 determines if all multi-view video footages are decoded into RGB frames and optical flow images. If any footage is not decoded, control returns to step 602 where MSBR system 130 selects the next video footage for decoding. By way of example, if video footage V 1 was previously selected at step 602 and more video footages are available for decoding (e.g., V2-V4), at this stage system 130 may select video segment V2 for decoding. If there is no remaining video footage to be decoded, method 504 ends.

[0111] The number of times system 130 executes steps 602-608 generally corresponds to the number of camera devices used for recording an event of interest. For example, if the recording setting is similar to Fig. 1 where four camera devices 110A-110D are used to record an event of interest, MSBR system 130 will execute each of steps 602-608 for 4 times. On the other hand, ifonly one camera representing one view is used for recording an event of interest, system 130 will only execute steps 602-608 once.

[0112] Turning to Fig. 7, a flowchart illustrating an example method 506 is described that is performed by MSBR system 130 in order to compute single-view prediction scores.

[0113] The method 506 commences at step 702 where MSBR system 130 selects one video footage representing one view of the event of interest. By way of example, at step 602 system 130 may select video footage VI as shown in the example above described in relation to step 502.

[0114] At step 704, MSBR system 310 selects a subset of RGB frames (previously created at step 604). For example, system 310 may select equally spaced RGB frames based on a prescribed sampling rate. It will be appreciated that other sampling techniques may be used or, instead, the complete set of RGB frames associated with video footage selected at step 702 may be used. In this disclosure, the sample set of RGB frames chosen at step 704 is denoted by M, the total number of RGB frames in the set is denoted by n. and each RGB frame in the set is denoted by Mi.

[0115] At step 706, MSBR system 310 predicts actions at frame level for all selected sample RGB frames. In one embodiment, system 310 may use action recognition algorithm in computer vision to recognise actions in a video footage based on RGB frames associated with the footage.

[0116] Generally, action recognition is a subfield of computer vision that deals with recognising human actions in videos. The main goal of an action recognition algorithm is to automatically identify and classify the actions performed by humans in video captures. There are several approaches to developing action recognition algorithms. For example, deep learningbased methods may be used for action recognition. This approach relies on deep neural networks, such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs), to learn features from video frames. It will be appreciated that in other embodiments other methods may be used.

[0117] In one embodiment, system 310 may use CNN-based action recognition algorithms. The set of RGB frames selected at step 704 are loaded into the CNN. The CNN provides prediction scores for the presence of actions (e.g., the existence of eye contact events) in each frame, which is denoted by F(Mi) or, more generally, as F(Mi;w) to represent the algorithm as a whole (i.e., afunction that takes the input images Mi and applies the CNN model, incorporating the weight adjustment denoted by W).

[0118] As the skilled reader will understand, the variable W represents the weights and biases used in the CNN in its inference stage (updated using standard backpropagation), as determined through the training.

[0119] By way of example, if footage VI is selected at step 702, and at least 5 RGB frames are selected at step 806 (M1-M5), then the predictions scores provided by the CNN at step 706 for the at least 5 frames of footage VI may be as follows:RGB frame Footage Eye contact presence No eye contact presenceM2VI 0.90 0.10M3VI 0.30 0.70M4VI 0.70 0.30M5VI 0.10 0.90

[0120] At step 708, MSBR system 310 computes a segment-level RGB consensus for the currently selected footage based on frame-level prediction scores. In one embodiment, the segment-level RGB consensus may be computed as an average of all prediction scores for all RGB frames of a footage. Continuing with the example described in the above table, RGB consensus scores for footage VI are as follows:Video Footage Eye contact presence No eye contact presence

[0121] At step 710, MSBR system 310 optionally selects a sample set of optical flow frames (previously created at step 606 and 608). In one embodiment, system 310 may randomly select consecutive optical flow images (Pi) of a specific length ( / ). It will be appreciated that other sampling techniques may be used or, instead, the complete set of optical flow images associated with video footage selected at steps 702 may be used.

[0122] At step 712, MSBR system 310 predicts actions for each sample set of optical flow images of a specific length (Pi,i). In one embodiment, system 310 may use action recognition algorithm in computer vision to recognise actions in a video footage based on optical flow images associated with the footage. As the reader will appreciate, the length I is selected on the basis of the annotated data from the model training.

[0123] In one embodiment, the present disclosure relies on convolutional neural networks (CNNs) for developing action recognition algorithms. The set of optical flow images selected at step 710 are loaded into the CNN. The CNN provides prediction scores for the presence of actions (e.g., the existence of eye contact events) in each set of flow images (Pi,i), which is denoted by F(Pi,i) or, more generally, as F(Pi,i;w) to represent the algorithm as a whole (i.e., a function that takes the input images Pi,i and applies the CNN model, incorporating the weight adjustment denoted by W).

[0124] As noted above, the variable W relates to the weights and biases used in the CNN in its inference stage, determined by way of the training. In other words, the same technique is applied for both the RGB and optical flow channels.

[0125] By way of example, if footage VI is selected at step 702, and at least 6 set of optical flow images with length of 4 (3 for x axis and 3 for y axis) are selected at step 710 (Pi,4-P6,4), then the predictions scores provided by the CNN at step 814 for the at least 6 set of optical flow images of segment 710 may be as follows:RGB frame Footage Eye contact presence No eye contact presencePi,4VI 0.81 0.19P2,4 VI 0.95 0.05P3~4 V I 032 0.68P4,4 VI 0.70 0.30P5,4 VI 0.10 0.90P6,4 VI 0.66 0.34

[0126] At step 714, MSBR system 310 computes a segment-level optical flow consensus for each footage based on optical flow image-level prediction scores. In one embodiment, segmentlevel optical flow consensus scores may be computed as an average of all prediction scores for all sets of optical flow images with a specific length (e.g., Pi,4-P6,4) of a segment. Continuing with the example described in the above table, optical flow consensus scores for footage V 1 may be as follows:Video footage Eye contact presence No eye contact presence

[0127] After segment-level RGB consensus and segment-level optical flow consensus scores are computed, method 506 continues to step 716.

[0128] At step 716, MSBR system 130 determines if all multi-view video footages are processed by method 506. If any footage is not processed, control returns to step 702 wheresystem 130 selects the next video footage for processing. By way of example, if video footage V 1 was previously selected at step 702 and more video footages are available for processing (e.g., V2-V4), at this stage system 130 may select video footage V2 for processing. If there is no remaining video footage to be processed, method 506 ends.

[0129] The number of times MSBR system 130 executes steps 702-714 corresponds to the number of camera devices used for recording an event of interest. For example, if the recording setting is similar to Fig. 1 where four camera devices 110A-110D are used to record an event of interest, system 130 will execute each of steps 702-714 for 4 times. On the other hand, if only one camera device representing one view is used for recording an event of interest, system 130 will only execute steps 702-714 once.

[0130] By way of example, assuming 4 views of an event of interest are recorded using camera devices 110, at the end of method 506 the resulting consensus scores may be as follows:Video Footage Category Eye contact presence No eye contact presenceVI RGB 0.56 0.44VI Optical flow 0.59 0.41"" -V4 Optical flow 0.25 0.75

[0131] Turning to Fig. 8, a flowchart illustrating an example method 508 is described that is performed by MSBR system 130 in order to aggregate prediction scores across multi-view video footages.

[0132] Method 508 commences at step 802 where MSBR system 130 selects one video footage representing one view of the event of interest. By way of example, at step 802 system 130 may select video footage VI.

[0133] At step 804, MSBR system 310 aggregates RGB prediction scores and optical flow prediction scores associated with the video footage selected at step 802 to produce fusion scores (MSBRFusion). In one embodiment, MSBRFusion scores may be computed by calculating the average of RGB and flow prediction scores.

[0134] At step 806, MSBR system 103 determines if all video footages (representing all views) associated with an event of interest are processed by method 508. If any footage is remaining,the control returns to step 802 where system 103 select the next footage for processing. If system 103 determines at step 804 that all footages are processed, control continues to step 808.

[0135] Continuing with the example in the previous table, at the end of step 806, MSBR system 130 may have computed the following MSBRFusion scores:Video Footage Eye contact presence No eye contact presenceVI 0.575 0.425V2 0.445 0.555V3 0.395 0.605

[0136] At step 808, MSBR system 130 computes final action prediction scores and determines if an event of interest includes eye contact events between the human participants. In one embodiment, computing the final action prediction scores carried out by calculating the average of the prediction scores across multiple views of an event of interest.

[0137] Continuing with the example shown in the above table, at step 808 MSBR system 310 may compute the following final scores across multi-view video footages:Eye contact presence No eye contact presence

[0138] At step 810, MSBR system 310 determines if eye contact is present in the event of interest. In one embodiment, system 310 makes the determination based on the prediction scores computed in step 808.

[0139] For example, MSBR system 310 may be programmed to classify an event of interest as containing eye contact if the eye contact presence score computed at step 810 is greater than the threshold score of 0.5. With this method of classification, and continuing with the example shown in the table above, the event of interest will be classified as not including eye contacts between the human subjects because the determined eye contact score of 0.16125 is not greater than the threshold of 0.5.Segmenting multi-view video footages

[0140] As described earlier, multi- view video footage preparation at step 502 of method 500 generally involve an additional step whereby MSBR system 130 divides each video footage (e.g. V1-V4) into shorter video segments.

[0141] By way of example, assume multi-view video footages are 20 minutes long. At step 502, system 310 may divide each video footage into multiple approximately 5-minute segments. The resulting set of video segments for one of video footages (e.g., VI) may, for example, be as shown in the following table.Video footage Camera Segment Start time End time 110A 1 10:00 AM 10:05 AM j();().AMIOFIO AM I lOA 3 10: 10 AM 10: 15 AM110A 4 10: 15 AM 10:20 AM

[0142] Video footages associated with other camera devices may be divided into shorter segments with matching timestamps as shown in the above table.

[0143] The example above (with 5 minute segments) is for illustration purposes only. Generally, to be of diagnostic value, the length of each video segments will be significantly shorter. For example, each video footage (e.g., VI) may be divided into segments of only a small number of successive frames, covering a fraction of a second or a few seconds.

[0144] Where there are several video segments per footage as described above, at step 504 of process 500 MSBR system 130 decodes all video segments of multi-view video footages into RGB frames and optical flow images as described earlier.

[0145] Similarly, at step 506 of process 500 MSBR system 130 computes single-view prediction scores based on several segments associated with one footage. To this end, system 310 carries out sampling steps 704 and 710 and action prediction steps 706 and 712 for each video segment.

[0146] Where there are several video segments per footage as described above, MSBR system 310 may compute segment-level RGB consensus scores across multiple footages (views). In some embodiments, system 310 may compute segment-level RGB consensus scores according to the following equation where m denotes the number of camera devices (e.g., m=4 matching devices 110), n denotes the number of RGB frames sampled at step 704, and function Hcdenotes the average consensus of input video segments from each camera:

[0147] Similarly, MSBR system 310 may compute segment-level optical flow consensus scores across multiple footages. In some embodiments, system 310 may compute segment-level optical flow consensus scores according to the following equation where where m denotes the number of camera devices (e.g., m=4 matching devices 110), function Hcdenotes the average consensus of input video segments from each camera, n denotes the number of optical flow image sets sampled at step 710, Pi denotes an optical flow image set of a specific length ( / ):

[0148] Furthermore, MSBR system 310 may compute aggregate prediction scores (MSBRFusion) across all camera views according to the following equation:MS B RFusion(Vsegment') = MS B RRGB(ysegment) ® MSBR Flowi segment)

[0149] At step 508 of method 500, system 310 determines if eye contact is present in the event of interest. In one embodiment, system 310 makes the determination based on the aggregate prediction scores (MSBRFusion) as described above.

[0150] Shorter video segments enable a finer analysis of the video footages. For example, MSBR system 130 may determine the occurrence of an eye contact event in a 2-second video segment. Accordingly, analysing 2-second segments associated with a 20-minute video footage (e.g. VI) may result in a series of approximately 600 pairs of MSBRRGB and MSBRFIOW scores. Such a series of scores may be further analysed for diagnostic purposes such as diagnosing autism in a child participant.

[0151] Turning to Fig. 9, a diagram depicting the flow of data and the processes of MSBR system 130 according to aspects of the present disclosure is presented. The input video from each camera angle is segmented into RGB frames (Mi), and grayscale optical flow image stacks (Pi) and fed into spatial (blue) and temporal (grey) channels respectively for training. The segment-level consensus scores from each single-modal of each input video clip are first calculated (bar graphs, FFiow[each_view] and FRGB[each_view]). Multi-modal predictions for all views of the same period are then aggregated separately through each channel. Multi-view late fusion (MSBRFusion(v_segment)) is then performed and the predictions from both channels (spatial and temporal) are averaged to produce a final prediction score on the presence or absence of eye contact behaviour.Data preparation and system training

[0152] The passages below describe a method of training the MSBR.

[0153] The free play task in the social interaction session is defined to begin when the adult assessor takes the relevant toys out of box and places them on the table or on the floor next to the child participant. After each session, a researcher records the start time of each task from each participant’s video recording.

[0154] In the tests carried out by the inventors, ffmpeg (version 4.2.4) library may be used to segment a standardised 5 -minute free play task from each longer video captures. Two independent coder persons used the MATLAB (version R2021a) Ground Truth Labeler application to annotate the start and end points of each eye contact event within the segmented videos.

[0155] An eye contact event is defined as the child participant directing visual attention to the eyes of the adult assessor while the assessor gazes into the child’s eyes. The free play task ends at the start of still face paradigm and reunion task (described below). If the still face paradigm occurred within the segmented videos, coder persons also marked the start time of its session and removed the annotations for this time period.

[0156] The still face paradigm is a research approach commonly used in developmental psychology to investigate social interaction and attachment, involving an interaction between an adult and a child, where the adult maintains a still and neutral facial expression, devoid of any emotional or responsive cues, while interacting with the child. The purpose is to observe the child's reactions and behaviours in response to the lack of social engagement or feedback from the adult. This paradigm helps researchers study the effects of disrupted social interactions on child development and attachment processes.

[0157] The reunion task is another known experimental approach, and follows the still face paradigm. In this task, after the period of still face interaction, the adult resumes normal social engagement and responsiveness towards the child. The purpose of the reunion task is to observe the child's reactions and behaviours upon the return of the social interaction, potentially revealing patterns of emotional regulation, attachment, or social engagement following the period of disruption in the still face paradigm.

[0158] Cohen's Kappa values were used to assess the inter-rater reliability of the two coder persons. All free play sessions for the non-autistic group and all such sessions for the autisticgroup were coded by both coder persons. The results showed moderate-to-high inter-rater reliability for annotations of direct eye contact. This included the kappa value for the non-autistic group (mean 0.72, standard deviation 0.25) and the kappa value for the autistic group (mean 0.84, standard deviation 0.15). The annotation reliability of the remaining sessions was further checked by the lead clinical researchers in the team.

[0159] The annotated free play segments were further trimmed into two categories: “Eye Contact” and “Other”, where “Other” refers to frames without any social interaction labels. The dataset was then prepared using a 5-fold cross validation and shuffled. A predetermined percentage of participant data (60%) in each fold was used to train the model, and the rest was used to validate and test the model (20% each). These percentages captured a unique set of participant data across the autistic and non-autistic groups within each fold, separately.

[0160] For frame-level training, the python library, DenseFlow, was used to decode each trimmed clip into RGB frames and optical flow images in the x and y axis directions. The dataset consists of 1,726 clips of eye contact events (42,110 RGB frames and 40,384 optical flow images in each axis direction) and 2,435 clips of “Other” events (3,849,732 RGB frames and 3,847,297 optical flow images in each axis direction).

[0161] The proposed method was implemented in PyTorch and executed on two 11GB NVIDIA GeForce RTX 2080Ti GPUs. All hyper-parameters tunning were decided using 5-fold cross-validation based on the performance of the validation sets. The best results were obtained when the learning rate was set to 3x10-5, the batch size was set to 10, the dropout ratio was set to 0.1. The spatial domain CNN branch was trained by extracting 10 RGB images per segment with equal interval of 1 (one) frame.

[0162] For the training of the time-domain CNN branch, the best performance was obtained when the learning rate was set to 8x10-6, the batch size was set to 32, the dropout rate was set to 0.6, and the input video was segmented in three clips, each clip containing four consecutive optical flow frames with an equal interval of 1 (one) frame.

[0163] In the tests carried out by the inventors, the system was initialised using the approach presented in Wang, Fimin, et al. "Temporal segment networks: Towards good practices for deep action recognition." (2016). Adaptive-moment-estimation (Kingma, Diederik P., and Jimmy Ba. "Adam: A method for stochastic optimization." (2014)) was used for network optimisation with CosineAnnealing learning rate scheduler due to its ability for fast coverage and less parameter-tunning. Standard back-propagation process was used to calculate the gradient of loss and to update model parameters (weight and biases) of the deep learning architecture. The training was set to stop at 300 epochs and used the standard categorical cross-entropy loss function.Test results

[0164] A total of 136 free play tasks (733.14 minutes) during social interaction sessions between a child and an adult assessor were annotated and analysed. The social interaction sessions involved 103 children diagnosed with autism and 23 children with no autism diagnosis. All children were aged 3 to 12 years old. All tasks were recorded with four high-definition (HD) camera devices mounted on the walls.

[0165] MSBR system 130 recognised eye contact events associated with 77 autistic children and 25 non-autistic children. The average duration of eye contact between child participants and the assessor during the free play task was 4.17 seconds (SD=6.80), and the average frequency of eye contact was 3.94 times (SD=5.27) across all included participants.

[0166] Descriptive statistics were calculated for age, gender and Nonverbal Intelligence Quotient (NVIQ) for the sample (see the below table). Independent-samples t tests showed no statistically significant differences in age (t(134)=1.245, p=0.215>0.05). However, statistically significant differences was found in NVIQ (t(70.895) = 7.088, p <0.001, two-sided), with 25 missing data points. There was statistically higher proportion of males to females which is expected given it was drawn from the autistic population that has higher prevalence for males (%2(1, N=136) = 4.976, p=0.026<0.05). autistic non-autisticDemographics Mean (SD) Mean (SD)N 103 33Gender (M / F) 87 / 16 22 / 11Age (years) 6.96 (2.72) 7.63 (2.67)NVIQ 89.18 (24.74) 116.04 (13.26)Data Set (Seconds)The Free Play Task Duration 344.55 (142.66) 257.57 (51.43)NVIQ = Nonverbal Developmental Quotient / Nonverbal Intelligence Quotient

[0167] The following table summarises the quantitative analysis of the average prediction performance of the MSBR system 130 for eye contact event recognition by single- view and multi- view video segments.

[0168] The overall performance of the model was evaluated by calculating the Top-1 accuracy of each single-view video segments for different modalities (spatial domain, temporal domain, and fusion of the two domains) separately on the test set. Precision, Recall and Fl scores were also reported.

[0169] Findings of the inventors show that eye contact behaviour of two social partners in a free play environment can be recognised in the view of panoramic camera devices fixed to the walls. Specifically, improvements were observed when a multi-modal average consensus considering both spatial and temporal features of the behaviour was fused for prediction. The findings are consistent irrespective of whether other participants are captured in multi-view video footages. For example, eye contact events between a child participant and an adult assessor can be recognised even when caregivers are sitting in the same room and captured in the multiview video footages.

[0170] Although the Fl score decreases by about 10% on optical flow, the consideration of temporal characteristics is necessary to study human motion patterns, and the multi-modal fusion process compensates well for this gap. Using multiple camera angles to record social interaction behaviours in the scene further improves the predictive performance of the MSBR system 310. Because of the stochastic nature of interactive behaviour in free play, capturing social behaviour from multiple camera angles provides more comprehensive visual information. These resultsshow promising preliminary evidence to provide a feasible solution for the recognition of multiple and diverse interactive behaviours in complex environments.

[0171] As will be understood from the above description, the present invention stems from the inventors’ surprising realisation that the coherence and complexity of human action mean that deep learning analysis of multi-modal continuous video data can be used to detect particular target behaviours of individuals, in particular eye contact, even when the eyes of the individual (or even the front of the individual’s face) are not visible in the data.

[0172] While the examples described above relate to assessment of possible learning disorders in children, and particularly ASD, it will be understood that the present invention has application to a variety of different health state conditions, such as assessing dementia in elderly persons, concussion effects or PTSD in trauma victims, or mental health states in patients (such as schizophrenia, Bipolar Disorder or chronic fatigue).

[0173] Unless otherwise stated, the terms “include” and “comprise” (and variations thereof such as “including”, “includes”, “comprising”, “comprises”, “comprised” and the like) are used inclusively and do not exclude further features, components, integers, steps, or elements.

[0174] Although the present disclosure uses terms “first,” “second,” etc. to describe various elements, these terms are used only to distinguish elements from one another and not in an ordinal sense.

[0175] It will be understood that the embodiments disclosed and defined in this specification extend to alternative combinations of two or more of the individual features mentioned in or evident from the text or drawings. All of these different combinations constitute alternative embodiments of the present disclosure.

[0176] The present specification describes various embodiments with reference to numerous specific details that may vary from implementation to implementation. No limitation, element, property, feature, advantage, or attribute that is not expressly recited in a claim should be considered as a required or essential feature. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense.

Claims

CLAIMS1. A method for use in assessing eye contact of a subject participant in a social interaction scenario between participants, the method involving: using camera apparatus to capture video images of said subject participant as a recorded image stream; processing said captured video images to provide data representing spatial and temporal features of the recorded image stream; and using a computer processing module to apply an action recognition algorithm to said data to predict occurrences of eye contact of said subject participant.

2. The method of claim 1, in which the prediction of occurrences of eye contact of said subject participant is made irrespective of whether the eyes of said subject participant are visible in one or more of the captured video images.

3. The method of claim 1 or claim 2, in which the subject participant is a child and a further participant is a facilitator, the social interaction scenario being a free play task adapted to assess social skills of the child interacting with the adult facilitator.

4. The method of any preceding claim, including using multiple camera apparatuses to capture video images of said subject participant from multiple angles.

5. The method of claim 4, wherein the multiple camera apparatus are positioned to collectively produce a panoramic view of the activities carried out by and the social interactions between the participants.

6. The method of any preceding claim, wherein the video images include or are associated with metadata including one or more of timestamp data and camera identifier data.

7. The method of any preceding claim, including determining prediction scores based on temporal properties of said recorded image stream.

8. The method of any preceding claim, including dividing the recorded image stream into a plurality of segments of prescribed length, and predicting occurrences of eye contact of said subject participant in each segment.

9. The method of any preceding claim, wherein the recorded image stream is decoded into two data sets:(a) a spatial frame data set, representing 2D spatial information from an array of pixels in a frame or a sequence of frames and(b) an optical flow data set, being a data set associated with flow vector patterns representing apparent motion of the content of an image between frames in a sequence of frames, the method including: analysing the spatial frame data set to obtain one or more first prediction scores associated with occurrence of eye contact of said subject participant, analysing the optical flow data set to obtain one or more second prediction scores associated with occurrence of eye contact of said subject participant; and combining said one or more first prediction scores and said one or more second prediction scores to obtain one or more aggregate prediction scores.

10. The method of claim 9, wherein the optical flow data set comprises a grayscale optical flow image stack.

11. The method of claim 9 or claim 10, including using multiple camera apparatuses to capture video images of said subject participant from multiple angles, wherein the respective spatial frame data sets and optical flow data sets are obtained for the recorded image stream from each camera apparatus and analysed independently to obtain consensus one or more first prediction scores and consensus one or more second prediction scores, such consensus scores being used to determine said one or more aggregate predictions scores.

12. The method of any preceding claim, including dividing the recorded image stream into a plurality of segments of prescribed length and predicting occurrences of eye contact of said subject participant in each segment, and predicting one or more aggregate predictions scores for each segment.

13. The method of claim 12, including analysing predicted occurrences of eye contact of said subject participant in multiple segments from the recorded image stream, in order to provide information useful for assessment one or more behavioural issues concerning the subject participant.

14. A method for use in training an action recognition algorithm to predict occurrences of eye contact of a subject participant in a social interaction scenario between participants, the method involving:processing multiple recorded image streams comprising captured video images of a number of subject participants in social interaction scenarios between participants, to provide data representing spatial and temporal features of each of the multiple recorded image streams; and classifying aspects of each of the multiple recorded image streams as representative or not of occurrences of eye contact of the subject participant in the respective image stream, for use in an eye contact prediction method.

15. The method of claim 14, wherein each of the multiple recorded image streams is decoded into two data sets:(a) a spatial frame data set, representing 2D spatial information from an array of pixels in a frame or a sequence of frames; and(b) an optical flow data set, being a data set associated with flow vector patterns representing apparent motion of the content of an image between frames in a sequence of frames, the method including classifying each spatial frame data set as representative or not of occurrence of eye contact of the subject participant in the image stream, classifying each optical flow data set as representative or not of occurrence of eye contact of the subject participant in the image stream, both said classifications for use in the eye contact prediction method.

Images